Move the default for PyPy to manylinux2014

[henryiii]

@mattip suggested moving to manylinux2014, since pypy is looking at dropping manylinux2010 in the future. cibuildwheel 2.3 (which will add pypy3.8) might be a good place to do it.

[henryiii]

One thought; when do we plan to move from manylinux2010 to manylinux2014 as the default for everything? manylinux2014 is already a much better choice (#901) for Python 3.10, NumPy doesn't even bother providing 2010 wheels for it, and that would avoid two docker launches if someone left the default alone. I think most users pick the manylinux image they want to use, anyway; usually because they want manylinux1.

[henryiii]

@joerick (and @mayeut and @YannickJadoul if you have any opinions here) what do you think? Should we bump PyPy to 2014 by default? And/or all Pythons? Due to the fact that 2014 is the best choice for Python 3.10, I'm mildly in favor of bumping all defaults to 2014.

[joerick]

I don't have a strong opinion on this, so I'd defer to @mattip - happy to bump the PyPy default to 2014.

One thought; when do we plan to move from manylinux2010 to manylinux2014 as the default for everything?

An excellent question, for which I don't have a firm answer on! I think it's fairly reasonable to lag here, since most packagers will normally want to build on the oldest (most compatible) image that works for their package (assuming the image is still maintained/secure).

With the default being something old, either they'll succeed on the first build, which gives them the most compatible wheel, or they'll keep trying newer and newer images until it works.

Do we fully understand the nature of the 3.10 problems you mention? I've not had any trouble building 3.10 wheels on manylinux2010.

[henryiii]

I think #901 was hitting an issue with manylinux2010, but mostly, since any Pip that supports Python 3.10 is new enough to understand manylinux2014, there's not much reason in using a past EOL distribution to build images. NumPy does not provide manylinux2010 wheels, only 2014, so anyone using NumPy will have to try to build NumPy from scratch, etc. If someone doesn't know what they are doing, they are more likely to hit issues with manylinux2010 than 2014 for Python 3.10.

[mattip]

Pinning 3.10 to manylinux2014 will take a bit of refactoring, since currently the pinned_docker_images.cfg only relates to platform, not to particular combinations of platform,python_version.

[henryiii]

I think we should just pin everything to 2014, simpler for consistency and users, and anyone wanting 2010 would just manually pin it, just like they do already with manylinux1. The slice of users serviced by 2010 is really pretty small; most users are stuck on pip 9 (manylinux1) or they have a pip from the last few years and are fine with manylinux2014. See the orange slices in pypa/manylinux#994 (comment) - Python 3.7 is the only one with a significant number of manylinux2010 as a maximum (3.6 is stuck with Pip 9 more often). The orange is always less than manylinux1, so it's less harsh than the manylinux1 -> manylinux2010 that we already default to.

[henryiii]

Another way to see it, https://mayeut.github.io/manylinux-timeline/ under "Policy readiness for python 3.*" plots. There's a small sliver for 2010 for 3.6, a wider but shrinking sliver for 2010 in 3.7, and it's simply too small to really see on any newer version.

With the default being something old, either they'll succeed on the first build, which gives them the most compatible wheel, or they'll keep trying newer and newer images until it works.

This is not how we handle our current selection; we've been on manylinux2010 for years; by this logic, we should set manylinux1 (and last I checked, 70% of our packages did exactly this). I think the logic is (and should be) we select a recommended default that a "basic" user should be happy with, and allow an advanced user to select whatever they want - if they really need extra compatibility, they can set manylinux2010 (or manylinux1, as they do now). A small, "normal" package should be happy with manylinux2014 (at least if they were with manylinux2010) - and manylinux2010 is based on a past-EOL operating system.

The reason for manylinux2014 is that it's now a significantly better choice for Python 3.10, but also, it's consistent - aarch64 and family all use 2014 already, since that's the earliest version that supported them, and now PyPy will be 2014 too; I think it's easier to explain if we just say 2014 is the default across the board. We even save a docker launch if pypy and cpython match.

[joerick]

I think the logic is (and should be) we select a recommended default that a "basic" user should be happy with, and allow an advanced user to select whatever they want - if they really need extra compatibility, they can set manylinux2010 (or manylinux1, as they do now).

Yeah, on reflection, this is a better way to think about this. I agree.

[Czaki]

Which python is shipped with pip which understands manylinux2014 tag? Many people do not upgrade pip after creating an environment.

[mattip]

When you say "shipped" what do you mean? The one on python.org (do linux users actually download and use that one?) or the one supplied by the distro? So then the question becomes multi-dimensional: which version of python on which distro on which platform

[Czaki]

Ok. I just checked that fur ubuntu 20.04 it is enough modern (20.x).
But default python installer for python 3.7 looks like is shipped with pip 19.2.3 which does not understand manylinux 2014.

[henryiii]

We supported Python 2.7 and Python 3.5 for years with manylinux2010 as the default, even though those shipped with much older pip's. In fact, Python 3.6's default pip can't access manylinux2010. And Python 3.7.8 has a pip new enough for manylinux2014. I think we should provide a "if you don't care, this is the most likely to work" solution, and right now 2014 is the simplest and the most likely to work for most users. Someone wanting to support older systems should select an older version and if they know to do that, hopefully they understand the consequences.

If we currently supported manylinux1, that would be different, that's a different "oldest possible" policy, but we don't.

There is a long discussion about manylinux and pip versions here: pypa/manylinux#994

[henryiii]

ubuntu 18.04 ships with pip 9, which can't access manylinux2010 already. Same with CentOS 7 and 8.

On Linux, the default installs don't matter, and no package manager cares what the default installer bundles anyway. :'( There are a lot of systems stuck on manylinux1 only Pip versions, but not many stuck on 2010 but not 2014 versions.

On macOS, you have to have a very modern pip to get any¹ wheels for macOS 11+. And very few are using the default installers there either. Also newer Pythons are common if you use the default installer, I don't think anyone is using an installer to get an ancient Python anyway.

On Windows, like macOS, the Python version tends to be pretty new.

Footnotes

1. Binary wheels, pure Python ones do work. ↩