Alternative wheel build

[0181532686cf4a31163be0bf3e6bb6732bf]

Hi,

I'm about to start working on adding support for non-glibc linux platforms (Alpine etc)[1, 2] to the pip/wheel toolset (pypa/manylinux#37). Long story short, pip wheels don't support musl libc at this moment, so if you want to install pandas in a neat small Alpine-based Docker image, you have to compile it from scratch, whether ubuntu/centos/etc mainstream distros are handled by binary wheels perfectly. And if you want to build package by yourself, you have to install compiler toolchain and spend some time on compilation, thus neglecting all possible benefits of using small Alpine-based image.

So, the question is how the binary management should be handled the best way: I'm planning to setup Travis CI job which will build wheel for pandas along with other popular packages (numpy, scipy, opencv to name a few) against musl libc, but this wheel should be published somewhere. What do you think is the most efficient and secure way of getting this wheel uploaded onto pypi registry? Thanks!

[1] https://alpinelinux.org
[2] http://www.etalabs.net/compare_libcs.html

[TomAugspurger]

Our wheel building infrastructure is currently in https://github.com/MacPython/pandas-wheels

We could add it there, or as a separate repo. Whatever is easiest for you probably. With that repo, the wheels are uploaded to wheels.scipy.org, from which the release manager downloads them and uploaded them to PyPI.

[0181532686cf4a31163be0bf3e6bb6732bf]

Thank you @TomAugspurger! I think separate repo would be better, but I will update you as soon as I will have something working.

[Bacto]

Hi,

Any news of this? It would be awesome to have prebuild binaries for Alpine Linux / Musl and I don't have enough skills to understand how pandas-wheels (https://github.com/MacPython/pandas-wheels) works.

It takes me 30 minutes and more than 1GB RAM to build Pandas :-\

[jreback]

@Bacto this would require community effort to do

maybe @matthew-brett knows of someone who already has this setup and we could copy

[matthew-brett]

The problem is that there is no specification for wheels on Alpine Linux / musl - I mean - there is no list of the minimum required libraries and libc versions. So I don't believe it's possible to build these wheels Iin any useful way).

[Bacto]

Yes @matthew-brett, I saw that problem and I finally think we need first specifications for wheels and Musl.
Thanks anyway for your reply :)

[adawalli]

The problem is that there is no specification for wheels on Alpine Linux / musl - I mean - there is no list of the minimum required libraries and libc versions. So I don't believe it's possible to build these wheels Iin any useful way).

Can you elaborate on this?

Pandas is grinding our CI/CD pipelines to nearly a halt after switching to alpine (we had to do this because the number of known CVE's against the base python debian images was just too shockingly high)

[TomAugspurger]

You can build a pandas wheel once and host it somewhere yourself. You just can't upload non-manylinux linux wheels to PyPI.

…

On Thu, Oct 10, 2019 at 7:23 AM adawalli ***@***.***> wrote: The problem is that there is no specification for wheels on Alpine Linux / musl - I mean - there is no list of the minimum required libraries and libc versions. So I don't believe it's possible to build these wheels Iin any useful way). Can you elaborate on this? Pandas is grinding our CI/CD pipelines to nearly a halt after switching to alpine (we had to do this because the number of known CVE's against the base python debian images was just too shockingly high) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22668?email_source=notifications&email_token=AAKAOIUPK3OO7VQKQS76UX3QN4NFFA5CNFSM4FUQPKJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA4BPLQ#issuecomment-540546990>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIQEEYLXX3XVHKXTNQLQN4NFFANCNFSM4FUQPKJQ> .

[matthew-brett]

To elaborate - if you want to be able to host Alpine wheels on PyPI - you'll need to go through the process of defining what compatibility the wheel should have, in terms of Alpine distributions, and in particular, what symbols and libraries it is compatible with - see https://www.python.org/dev/peps/pep-0513/ for the original specification for glibc Intel Linux. Once that's agreed, then you would need to get that specification checked and included in pip, and you'd need something like auditwheel to fix up your wheels. So Tom's suggestion is a better one for you, probably; just build the wheels that work for your CI machines, and host them somewhere other than PyPI.

[adawalli]

Thanks @TomAugspurger and @matthew-brett - I already use quite a bit of custom caching and was hoping to keep things as simple as possible, while still using pyproject.toml without any custom repo paths, but I will certainly re-evaluate that. I appreciate you taking the time to respond!

[TomAugspurger]

I don't think there's anything pandas can do with this at the moment. If / when there's a PEP for other platforms and it's added to multibuild then we'll add it into our build setup at https://github.com/macpython/pandas-wheels.

[sanmai-NL]

@TomAugspurger musllinux wheels are available for many packages on PyPI. What currently holding this up for Pandas?

[TomAugspurger]

https://peps.python.org/pep-0656/ is accepted now, and I think that cibuildwheel supports building musllinux wheels now. So probably just someone to do the work.

I'd recommend verifying that all our dependencies have wheels first, though. Otherwise the value of a pandas musllinux wheel is limited.

[sanmai-NL]

That's creating a bottleneck isn't it. Actually, saving compilation time for Pandas even if some of it deps need to be built is valuable.