gh-90548: Make musl test skips smarter (fixes Alpine errors) #131313

bitdancer · 2025-03-15T22:31:22Z

Make musl test skips smarter (fixes Alpine errors)

A relatively small number of tests fail when the underlying c library is
provided by musl. This was originally reported in bpo-46390 by
Christian Heimes. Among other changes, these tests were marked for
skipping in gh-31947/ef1327e3 as part of bpo-40280 (emscripten support),
but the skips were conditioned on the platform being emscripten (or
wasi, skips for which ere added in 9b50585).

In gh-131071 Victor Stinner added a linked_to_musl function to enable
skipping a test in test_math that fails under musl, like it does on a
number of other platforms. This check can successfully detect that
python is running under musl on Alpine, which was the original problem
report in bpo-46390.

This PR replaces Victor's solution with an enhancement to
platform.libc_ver that does the check more cheaply, and also gets the
version number. The latter is important because the math test being
skipped is due to a bug in musl that has been fixed, but as of this
checkin date has not yet been released. When it is, the test skip can
be fixed to check for the minimum needed version.

The enhanced version of linked_to_musl is also used to do the skips of
the other tests that generically fail under musl, as opposed to
emscripten or wasi only failures. This will allow these tests to be
skipped automatically on Alpine.

This PR does not enhance libc_ver to support emscripten and wasi, as
I'm not familiar with those platforms; instead it returns a version
triple of (0, 0, 0) for those platforms. This means the musl tests will
be skipped regardless of musl version, so ideally someone will add
support to libc_ver for these platforms.

📚 Documentation preview 📚: https://cpython-previews--131313.org.readthedocs.build/

Issue: Multiple test failures on Alpine 3.15 / musl-1.2.2-r7 #90548

A relatively small number of tests fail when the underlying c library is provided by musl. This was originally reported in bpo-46390 by Christian Heimes. Among other changes, these tests were marked for skipping in pythongh-31947/ef1327e3 as part of bpo-40280 (emscripten support), but the skips were conditioned on the *platform* being emscripten (or wasi, skips for which ere added in 9b50585). In pythongh-131071 Victor Stinner added a linked_to_musl function to enable skipping a test in test_math that fails under musl, like it does on a number of other platforms. This check can successfully detect that python is running under musl on Alpine, which was the original problem report in bpo-46390. This PR replaces Victor's solution with an enhancement to platform.libc_ver that does the check more cheaply, and also gets the version number. The latter is important because the math test being skipped is due to a bug in musl that has been fixed, but as of this checkin date has not yet been released. When it is, the test skip can be fixed to check for the minimum needed version. The enhanced version of linked_to_musl is also used to do the skips of the other tests that generically fail under musl, as opposed to emscripten or wasi only failures. This will allow these tests to be skipped automatically on Alpine. This PR does *not* enhance libc_ver to support emscripten and wasi, as I'm not familiar with those platforms; instead it returns a version triple of (0, 0, 0) for those platforms. This means the musl tests will be skipped regardless of musl version, so ideally someone will add support to libc_ver for these platforms.

In adding tests for the new platform code I found a bug in the old code: if a valid version is passed for version and it is greater than the version found for an so *and* there is no glibc version, then the version from the argument was returned. The code changes here fix that, as well as fixing my own broken additions.

Doc/library/test.rst

Lib/test/test_math.py

Lib/test/test_os.py

vstinner · 2025-03-17T17:37:47Z

Lib/test/test_platform.py

@@ -552,6 +552,7 @@ def test_libc_ver(self):
                (b'GLIBC_2.9', ('glibc', '2.9')),
                (b'libc.so.1.2.5', ('libc', '1.2.5')),
                (b'libc_pthread.so.1.2.5', ('libc', '1.2.5_pthread')),
+                (b'/aports/main/musl/src/musl-1.2.5', ('musl', '1.2.5')),


Can you add tests on version with 2 numbers and 4 numbers? Like /aports/main/musl/src/musl-1.2 and /aports/main/musl/src/musl-1.2.5.1.

I could, but I don't believe musl will ever have such release numbers, so is it worth doing?

Parsing a binary file using a regular expression is a strange thing for me. So I prefer to have tests to make sure that the implementation is reliable. The question is more if the regex will match versions with 2 members (x.y) or 4 members (x.y.z.a), or if it only matchs version with 3 members (x.y.z).

Fair enough, though the existing tests don't do that, nor do they test the no-match cases. I could add such tests, but at that point I'd want to rewrite this test method to split it up into multiple tests, and I don't think that's in scope for this PR :) I'll add the couple of extra checks, though.

Lib/test/test_support.py

vstinner · 2025-03-17T17:40:08Z

Lib/test/test_support.py

@@ -746,7 +746,18 @@ def test_get_signal_name(self):

    def test_linked_to_musl(self):
        linked = support.linked_to_musl()
-        self.assertIsInstance(linked, bool)
+        self.assertIsNotNone(linked)
+        if support.is_wasi:


What about is_emscripten?

I used is_wasi as an example, but it doesn't hurt to add emscripten as well.

Lib/test/test_support.py

vstinner · 2025-03-17T17:44:52Z

The code detecting the musl version works for me on Alpine Linux:

~/cpython # LD_LIBRARY_PATH=$PWD ./python 
Python 3.14.0a5+ (heads/main-dirty:fe186d76cfd, Mar 11 2025, 08:54:51) [GCC 14.2.0] on linux
>>> import test.support
>>> test.support.linked_to_musl()
(1, 2, 5)


~/cpython # LD_LIBRARY_PATH=$PWD ./python -m test.pythoninfo|grep libc
platform.libc_ver: musl 1.2.5


~/cpython # apk info musl
musl-1.2.5-r9 description:
the musl c library (libc) implementation

musl-1.2.5-r9 webpage:
https://musl.libc.org/

musl-1.2.5-r9 installed size:
646 KiB

picnixz

For my suggestions with dedents, please check them manually as I'm not entirely sure that I didn't mix tabs and spaces.

Lib/platform.py

Lib/test/support/__init__.py

picnixz · 2025-03-17T17:48:08Z

Lib/test/test_os.py

+    @unittest.skipIf(
+        support.linked_to_musl(),
+        'musl pathconf ignores the file descriptor and returns a constant',
+        )


Suggested change

)

)

I prefer the style I used because it lines up better with how python's statements work (first line only is outdented, lack of outdent signals return to outer logic block). This is accepted by PEP 8, are there additional style constrains on the cpython code base these days?

I haven't seen lots of use of hanging indents in the lib. There is no additional constraints but usually we try to be consistent with the surrounding code when possible.

Lib/test/test_platform.py

Lib/test/test_support.py

Lib/test/test_platform.py

picnixz · 2025-03-17T17:52:40Z

Btw, do we want to create an issue (maybe an umbrella one for musl issues) or do we make it a skip-issue?

bitdancer · 2025-03-17T18:11:23Z

There is an issue (gh-90548). I forgot the gh issue existed when I mentioned the bpo issue :( I'm not actually sure how one does the link, as I'm still coming up to speed on the "modern" way of doing development ;)

picnixz · 2025-03-17T18:16:38Z

I'm not actually sure how one does the link, as I'm still coming up to speed on the "modern" way of doing development ;)

No worries! the PR title should be gh-XXXXXX: title or GH-XXXXXX: title and the bot would link it.

vstinner

LGTM, but please fix the important typo 'musl' in binary.

vstinner · 2025-03-18T10:00:39Z

Lib/platform.py

    with open(executable, 'rb') as f:
        binary = f.read(chunksize)
        pos = 0
        while pos < len(binary):
-            if b'libc' in binary or b'GLIBC' in binary:
+            if b'libc' in binary or b'GLIBC' or 'musl' in binary:


Suggested change

if b'libc' in binary or b'GLIBC' or 'musl' in binary:

if b'libc' in binary or b'GLIBC' or b'musl' in binary:

Oh, that's a nasty one. The actual fix is b'GLIBC in binary or b'musl' in binary`. That's why it passed without the b...it just ran slower ;) (A lot slower, somewhat to my surprise.)

picnixz

As I said, I won't block the PR for cosmetics only though I would have aligned with the surrounding/existing code when possible.

serhiy-storchaka

LGTM in general.

serhiy-storchaka · 2025-03-18T11:09:45Z

Lib/platform.py

+    libc_search = re.compile(
+        br'(__libc_init)'
+        br'|'
+        br'(GLIBC_([0-9.]+))'
+        br'|'
+        br'(libc(_\w+)?\.so(?:\.(\d[0-9.]*))?)'
+        br'|'
+        br'(musl-([0-9.]+))'
+        br'',
+        re.ASCII)


BTW, we can switch to the VERBOSE mode, which would be slightly more clear.

Suggested change

libc_search = re.compile(

br'(__libc_init)'

br'|'

br'(GLIBC_([0-9.]+))'

br'|'

br'(libc(_\w+)?\.so(?:\.(\d[0-9.]*))?)'

br'|'

br'(musl-([0-9.]+))'

br'',

re.ASCII)

libc_search = re.compile(br'''

(__libc_init)

| (GLIBC_([0-9.]+))

| (libc(_\w+)?\.so(?:\.(\d[0-9.]*))?)

| (musl-([0-9.]+))

''',

re.ASCII | re.VERBOSE)

I like that.

serhiy-storchaka · 2025-03-18T11:15:25Z

Lib/test/support/__init__.py

+    # emscripten (at least as far as we're concerned) and wasi use musl,
+    # but platform doesn't know how to get the version, so set it to zero.
+    if is_emscripten or is_wasi:
+        return (_linked_to_musl := (0, 0, 0))


I do not think that it is necessary to use the := operator here.

It is never necessary to use the := operator, so do you mean you prefer the more verbose style (which I'm fine with changing to), or am I missing something?

I prefer the more verbose style, which is also more sequential and clear.

serhiy-storchaka · 2025-03-18T11:23:09Z

Lib/test/test_strptime.py

-        support.is_emscripten or support.is_wasi,
-        "musl libc issue on Emscripten, bpo-46390"
-    )
+    @unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390")
    @run_with_locales('LC_TIME', 'en_US', 'fr_FR', 'de_DE', 'ja_JP',


We can add the linked_to_musl() check in run_with_locales() and run_with_locale(). This may be non-trivial, depending on arguments, a dummy run can be supported even if no suitable locale was found. For now, the proposed change is good as is.

This didn't cause a test failure because the broken logic caused the regex to always run...which is a lot slower.

bitdancer · 2025-03-19T17:05:59Z

I'm thinking this is not worth backporting?

vstinner · 2025-03-19T17:07:25Z

I'm thinking this is not worth backporting?

I don't think that it's worth it to backport this change to stable branches. Supporting Alpine Linux can be seen as a new feature ;-)

bitdancer added 3 commits March 15, 2025 15:17

Add docs, including some existing is_xxx's.

417143e

bitdancer requested a review from pganssle as a code owner March 15, 2025 22:31

bitdancer self-assigned this Mar 15, 2025

bitdancer requested a review from abalkin as a code owner March 15, 2025 22:31

bedevere-app bot added the awaiting core review label Mar 15, 2025

bitdancer changed the title ~~better musl test checks~~ Make musl test skips smarter (fixes Alpine errors) Mar 15, 2025

bitdancer requested review from vstinner and tiran March 15, 2025 22:40

bitdancer mentioned this pull request Mar 17, 2025

Multiple test failures on Alpine 3.15 / musl-1.2.2-r7 #90548

Open

vstinner mentioned this pull request Mar 17, 2025

test_re fails on Alpine Linux (with musl) #131342

Closed

vstinner reviewed Mar 17, 2025

View reviewed changes

vstinner added the skip news label Mar 17, 2025

picnixz reviewed Mar 17, 2025

View reviewed changes

picnixz changed the title ~~Make musl test skips smarter (fixes Alpine errors)~~ gh-90548: Make musl test skips smarter (fixes Alpine errors) Mar 17, 2025

bitdancer added 3 commits March 17, 2025 14:24

Address vstinner review comments.

ef36eb2

Test some version number variants we accept.

909ea3b

Add news item about libc_ver enhancement.

4e76d78

vstinner approved these changes Mar 18, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Mar 18, 2025

picnixz approved these changes Mar 18, 2025

View reviewed changes

serhiy-storchaka approved these changes Mar 18, 2025

View reviewed changes

bitdancer added 2 commits March 18, 2025 14:49

Fix 'in' phrase logic.

83add5d

This didn't cause a test failure because the broken logic caused the regex to always run...which is a lot slower.

Prettify re expression using re.VERBOSE.

4d6aa92

Use more verbose style rather than :=

111e2ec

serhiy-storchaka approved these changes Mar 19, 2025

View reviewed changes

bitdancer merged commit 6146295 into python:main Mar 19, 2025
39 checks passed

bedevere-app bot removed the awaiting merge label Mar 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-90548: Make musl test skips smarter (fixes Alpine errors) #131313

gh-90548: Make musl test skips smarter (fixes Alpine errors) #131313

bitdancer commented Mar 15, 2025 •

edited by bedevere-app bot

Loading

vstinner Mar 17, 2025

bitdancer Mar 17, 2025

vstinner Mar 17, 2025

bitdancer Mar 17, 2025

vstinner Mar 17, 2025

bitdancer Mar 17, 2025

vstinner commented Mar 17, 2025

picnixz left a comment

picnixz Mar 17, 2025

bitdancer Mar 17, 2025

picnixz Mar 17, 2025

picnixz commented Mar 17, 2025

bitdancer commented Mar 17, 2025

picnixz commented Mar 17, 2025

vstinner left a comment

vstinner Mar 18, 2025

bitdancer Mar 18, 2025

picnixz left a comment

serhiy-storchaka left a comment

serhiy-storchaka Mar 18, 2025

bitdancer Mar 18, 2025

serhiy-storchaka Mar 18, 2025

bitdancer Mar 18, 2025

serhiy-storchaka Mar 18, 2025

bitdancer Mar 18, 2025

serhiy-storchaka Mar 18, 2025

bitdancer commented Mar 19, 2025

vstinner commented Mar 19, 2025

	if b'libc' in binary or b'GLIBC' or 'musl' in binary:
	if b'libc' in binary or b'GLIBC' or b'musl' in binary:

gh-90548: Make musl test skips smarter (fixes Alpine errors) #131313

gh-90548: Make musl test skips smarter (fixes Alpine errors) #131313

Conversation

bitdancer commented Mar 15, 2025 • edited by bedevere-app bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vstinner commented Mar 17, 2025

picnixz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

picnixz commented Mar 17, 2025

bitdancer commented Mar 17, 2025

picnixz commented Mar 17, 2025

vstinner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

picnixz left a comment

Choose a reason for hiding this comment

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitdancer commented Mar 19, 2025

vstinner commented Mar 19, 2025

bitdancer commented Mar 15, 2025 •

edited by bedevere-app bot

Loading