Python 3.11

[filmor]

• Add Python 3.11 type offsets
• Add Python 3.11 to metadata and workflows
• Improve geninterop script to handle new case in 3.11
• Fix offsets for 3.11

What does this implement/fix? Explain your changes.

...

Does this close any currently open issues?

...

Any other comments?

...

Checklist

Check all those that are applicable and complete.

• Make sure to include one or more tests for your change
• If an enhancement PR, please create docs and at best an example
• Ensure you have signed the .NET Foundation CLA
• Add yourself to AUTHORS
• Updated the CHANGELOG

[lostmsu]

This creates multiple sources of truth. Does editing post creation not work?

[filmor]

It doesn't. Python 3.11 rejects creating a type that has HAVE_GC set but not these slots. Rightly so :)

[lostmsu]

Can we have this array shared with the other place that used to use this function?

[lostmsu]

I can take a look once 3.11 hits Anaconda.

[lostmsu]

Re only clear on tp_dictoffset > 0. What is the scenario when it does not hold? I suspect the assert might be there to ensure something did not blow up earlier.

[filmor]

Re only clear on tp_dictoffset > 0. What is the scenario when it does not hold? I suspect the assert might be there to ensure something did not blow up earlier.

tp_dictoffset is only > 0 if there is actually a tp_dict, which is apparently not the case anymore for a lot of classes in Python 3.11. Fixing this fixed almost all segfaults (which were probably just unhandled .NET exceptions in the end). The only one left is the derived class in RecursiveInheritance requiring a similar setup for the slots (i.e. up-front instead of via modification) as I did in the other case. Here, it complains about tp_traverse.

[lostmsu]

OK, just wanted to ensure you investigated. Any specific type combinations that turn out to be an issue to make a test?

[filmor]

The only issue left is fixing (and reenabling tests) for Py_SetPythonHome (which is actually deprecated since 3.11). I'm not quite sure what's going on here, but something seems to fail in the encoding as it assumes in the test that the path is \0xfffd, so Unicode Replacement Character.

[filmor]

The remaining issue seems to be precisely calling Py_SetPythonHome with an empty string, which seems to corrupt the underlying storage. I'll investigate a bit more, but in the worst case, we should probably just forbid it.

[lostmsu]

LGTM, but I'll wait for the CI passing.

[filmor]

Wow, from my PoV this is actually a bug in Python itself:

https://github.com/python/cpython/blob/c0859743d9ad3bbd4c021200f4162cfeadc0c17a/Python/pathconfig.c#L255-L273

If Py_SetPythonHome was called with a non-empty string and is subsequently called with an empty one, it will always run PyMem_RawFree but only actually reset the pointer in .home if has_value is set, so if home && home[0] (i.e. non-empty string).

I'll try to create a proper bug-report for this, but for now I will just forbid setting PythonHome to an empty string. All of the other property setters have the same issue, AFAICT.

The issue occurs in our test-suite, because we usually start up with an empty PYTHONHOME, and then we try to update and later reset it. I don't know why this only occurs in Python 3.11, but the bug as such exists before that as well.

[filmor]

I have investigated the issue further: It's indeed a regression in Python 3.11, as soon as it's fixed up-stream, I'll adjust our tests to always run again.