Merge branch 'main' into pythongh-97822

Author: JelleZijlstra

Doc/faq/design.rst

@@ -129,7 +129,7 @@ reference or call the method from a particular class.  In C++, if you want to
 use a method from a base class which is overridden in a derived class, you have
 to use the ``::`` operator -- in Python you can write
 ``baseclass.methodname(self, <argument list>)``.  This is particularly useful
-for :meth:`__init__` methods, and in general in cases where a derived class
+for :meth:`~object.__init__` methods, and in general in cases where a derived class
 method wants to extend the base class method of the same name and thus has to
 call the base class method somehow.
 
@@ -232,7 +232,8 @@ Similar methods exist for bytes and bytearray objects.
 How fast are exceptions?
 ------------------------
 
-A try/except block is extremely efficient if no exceptions are raised.  Actually
+A :keyword:`try`/:keyword:`except` block is extremely efficient if no exceptions
+are raised.  Actually
 catching an exception is expensive.  In versions of Python prior to 2.0 it was
 common to use this idiom::
 
@@ -352,7 +353,7 @@ will probably run out of file descriptors::
        c = f.read(1)
 
 Indeed, using CPython's reference counting and destructor scheme, each new
-assignment to *f* closes the previous file.  With a traditional GC, however,
+assignment to ``f`` closes the previous file.  With a traditional GC, however,
 those file objects will only get collected (and closed) at varying and possibly
 long intervals.
 
@@ -376,10 +377,10 @@ Python to work with it.)
 
 Traditional GC also becomes a problem when Python is embedded into other
 applications.  While in a standalone Python it's fine to replace the standard
-malloc() and free() with versions provided by the GC library, an application
-embedding Python may want to have its *own* substitute for malloc() and free(),
+``malloc()`` and ``free()`` with versions provided by the GC library, an application
+embedding Python may want to have its *own* substitute for ``malloc()`` and ``free()``,
 and may not want Python's.  Right now, CPython works with anything that
-implements malloc() and free() properly.
+implements ``malloc()`` and ``free()`` properly.
 
 
 Why isn't all memory freed when CPython exits?
@@ -401,14 +402,15 @@ Why are there separate tuple and list data types?
 
 Lists and tuples, while similar in many respects, are generally used in
 fundamentally different ways.  Tuples can be thought of as being similar to
-Pascal records or C structs; they're small collections of related data which may
+Pascal ``records`` or C ``structs``; they're small collections of related data which may
 be of different types which are operated on as a group.  For example, a
 Cartesian coordinate is appropriately represented as a tuple of two or three
 numbers.
 
 Lists, on the other hand, are more like arrays in other languages.  They tend to
 hold a varying number of objects all of which have the same type and which are
-operated on one-by-one.  For example, ``os.listdir('.')`` returns a list of
+operated on one-by-one.  For example, :func:`os.listdir('.') <os.listdir>`
+returns a list of
 strings representing the files in the current directory.  Functions which
 operate on this output would generally not break if you added another file or
 two to the directory.
@@ -444,9 +446,9 @@ far) under most circumstances, and the implementation is simpler.
 
 Dictionaries work by computing a hash code for each key stored in the dictionary
 using the :func:`hash` built-in function.  The hash code varies widely depending
-on the key and a per-process seed; for example, "Python" could hash to
--539294296 while "python", a string that differs by a single bit, could hash
-to 1142331976.  The hash code is then used to calculate a location in an
+on the key and a per-process seed; for example, ``'Python'`` could hash to
+``-539294296`` while ``'python'``, a string that differs by a single bit, could hash
+to ``1142331976``.  The hash code is then used to calculate a location in an
 internal array where the value will be stored.  Assuming that you're storing
 keys that all have different hash values, this means that dictionaries take
 constant time -- O(1), in Big-O notation -- to retrieve a key.
@@ -497,7 +499,8 @@ Some unacceptable solutions that have been proposed:
 
 There is a trick to get around this if you need to, but use it at your own risk:
 You can wrap a mutable structure inside a class instance which has both a
-:meth:`__eq__` and a :meth:`__hash__` method.  You must then make sure that the
+:meth:`~object.__eq__` and a :meth:`~object.__hash__` method.
+You must then make sure that the
 hash value for all such wrapper objects that reside in a dictionary (or other
 hash based structure), remain fixed while the object is in the dictionary (or
 other structure). ::
@@ -528,7 +531,7 @@ is True``) then ``hash(o1) == hash(o2)`` (ie, ``o1.__hash__() == o2.__hash__()``
 regardless of whether the object is in a dictionary or not.  If you fail to meet
 these restrictions dictionaries and other hash based structures will misbehave.
 
-In the case of ListWrapper, whenever the wrapper object is in a dictionary the
+In the case of :class:`!ListWrapper`, whenever the wrapper object is in a dictionary the
 wrapped list must not change to avoid anomalies.  Don't do this unless you are
 prepared to think hard about the requirements and the consequences of not
 meeting them correctly.  Consider yourself warned.
@@ -581,9 +584,9 @@ exhaustive test suites that exercise every line of code in a module.
 An appropriate testing discipline can help build large complex applications in
 Python as well as having interface specifications would.  In fact, it can be
 better because an interface specification cannot test certain properties of a
-program.  For example, the :meth:`append` method is expected to add new elements
+program.  For example, the :meth:`list.append` method is expected to add new elements
 to the end of some internal list; an interface specification cannot test that
-your :meth:`append` implementation will actually do this correctly, but it's
+your :meth:`list.append` implementation will actually do this correctly, but it's
 trivial to check this property in a test suite.
 
 Writing test suites is very helpful, and you might want to design your code to
@@ -599,14 +602,14 @@ Why is there no goto?
 In the 1970s people realized that unrestricted goto could lead
 to messy "spaghetti" code that was hard to understand and revise.
 In a high-level language, it is also unneeded as long as there
-are ways to branch (in Python, with ``if`` statements and ``or``,
-``and``, and ``if-else`` expressions) and loop (with ``while``
-and ``for`` statements, possibly containing ``continue`` and ``break``).
+are ways to branch (in Python, with :keyword:`if` statements and :keyword:`or`,
+:keyword:`and`, and :keyword:`if`/:keyword:`else` expressions) and loop (with :keyword:`while`
+and :keyword:`for` statements, possibly containing :keyword:`continue` and :keyword:`break`).
 
 One can also use exceptions to provide a "structured goto"
 that works even across
 function calls.  Many feel that exceptions can conveniently emulate all
-reasonable uses of the "go" or "goto" constructs of C, Fortran, and other
+reasonable uses of the ``go`` or ``goto`` constructs of C, Fortran, and other
 languages.  For example::
 
    class label(Exception): pass  # declare a label
@@ -620,7 +623,7 @@ languages.  For example::
    ...
 
 This doesn't allow you to jump into the middle of a loop, but that's usually
-considered an abuse of goto anyway.  Use sparingly.
+considered an abuse of ``goto`` anyway.  Use sparingly.
 
 
 Why can't raw strings (r-strings) end with a backslash?
@@ -652,7 +655,7 @@ If you're trying to build a pathname for a DOS command, try e.g. one of ::
 Why doesn't Python have a "with" statement for attribute assignments?
 ---------------------------------------------------------------------
 
-Python has a 'with' statement that wraps the execution of a block, calling code
+Python has a :keyword:`with` statement that wraps the execution of a block, calling code
 on the entrance and exit from the block.  Some languages have a construct that
 looks like this::
 
@@ -679,13 +682,13 @@ For instance, take the following incomplete snippet::
        with a:
            print(x)
 
-The snippet assumes that "a" must have a member attribute called "x".  However,
+The snippet assumes that ``a`` must have a member attribute called ``x``. However,
 there is nothing in Python that tells the interpreter this. What should happen
-if "a" is, let us say, an integer?  If there is a global variable named "x",
-will it be used inside the with block?  As you see, the dynamic nature of Python
+if ``a`` is, let us say, an integer?  If there is a global variable named ``x``,
+will it be used inside the :keyword:`with` block?  As you see, the dynamic nature of Python
 makes such choices much harder.
 
-The primary benefit of "with" and similar language features (reduction of code
+The primary benefit of :keyword:`with` and similar language features (reduction of code
 volume) can, however, easily be achieved in Python by assignment.  Instead of::
 
    function(args).mydict[index][index].a = 21
@@ -714,7 +717,8 @@ Why don't generators support the with statement?
 For technical reasons, a generator used directly as a context manager
 would not work correctly.  When, as is most common, a generator is used as
 an iterator run to completion, no closing is needed.  When it is, wrap
-it as "contextlib.closing(generator)" in the 'with' statement.
+it as :func:`contextlib.closing(generator) <contextlib.closing>`
+in the :keyword:`with` statement.
 
 
 Why are colons required for the if/while/def/class statements?

Doc/library/urllib.parse.rst

@@ -113,7 +113,8 @@ or on combining URL components into a URL string.
    +------------------+-------+-------------------------+------------------------+
    | :attr:`path`     | 2     | Hierarchical path       | empty string           |
    +------------------+-------+-------------------------+------------------------+
-   | :attr:`params`   | 3     | No longer used          | always an empty string |
+   | :attr:`params`   | 3     | Parameters for last     | empty string           |
+   |                  |       | path element            |                        |
    +------------------+-------+-------------------------+------------------------+
    | :attr:`query`    | 4     | Query component         | empty string           |
    +------------------+-------+-------------------------+------------------------+

Doc/tutorial/classes.rst

@@ -581,7 +581,8 @@ this::
        .
        <statement-N>
 
-The name :class:`BaseClassName` must be defined in a scope containing the
+The name :class:`BaseClassName` must be defined in a
+namespace accessible from the scope containing the
 derived class definition.  In place of a base class name, other arbitrary
 expressions are also allowed.  This can be useful, for example, when the base
 class is defined in another module::

Lib/asyncio/selector_events.py

@@ -58,6 +58,7 @@ def __init__(self, selector=None):
 
     def _make_socket_transport(self, sock, protocol, waiter=None, *,
                                extra=None, server=None):
+        self._ensure_fd_no_transport(sock)
         return _SelectorSocketTransport(self, sock, protocol, waiter,
                                         extra, server)
 
@@ -68,6 +69,7 @@ def _make_ssl_transport(
             ssl_handshake_timeout=constants.SSL_HANDSHAKE_TIMEOUT,
             ssl_shutdown_timeout=constants.SSL_SHUTDOWN_TIMEOUT,
     ):
+        self._ensure_fd_no_transport(rawsock)
         ssl_protocol = sslproto.SSLProtocol(
             self, protocol, sslcontext, waiter,
             server_side, server_hostname,
@@ -80,6 +82,7 @@ def _make_ssl_transport(
 
     def _make_datagram_transport(self, sock, protocol,
                                  address=None, waiter=None, extra=None):
+        self._ensure_fd_no_transport(sock)
         return _SelectorDatagramTransport(self, sock, protocol,
                                           address, waiter, extra)
 

Lib/re/__init__.py

@@ -229,6 +229,7 @@ def compile(pattern, flags=0):
 def purge():
     "Clear the regular expression caches"
     _cache.clear()
+    _cache2.clear()
     _compile_repl.cache_clear()
 
 def template(pattern, flags=0):
@@ -266,40 +267,64 @@ def escape(pattern):
 # --------------------------------------------------------------------
 # internals
 
-_cache = {}  # ordered!
-
+# Use the fact that dict keeps the insertion order.
+# _cache2 uses the simple FIFO policy which has better latency.
+# _cache uses the LRU policy which has better hit rate.
+_cache = {}  # LRU
+_cache2 = {}  # FIFO
 _MAXCACHE = 512
+_MAXCACHE2 = 256
+assert _MAXCACHE2 < _MAXCACHE
+
 def _compile(pattern, flags):
     # internal: compile pattern
     if isinstance(flags, RegexFlag):
         flags = flags.value
     try:
-        return _cache[type(pattern), pattern, flags]
+        return _cache2[type(pattern), pattern, flags]
     except KeyError:
         pass
-    if isinstance(pattern, Pattern):
-        if flags:
-            raise ValueError(
-                "cannot process flags argument with a compiled pattern")
-        return pattern
-    if not _compiler.isstring(pattern):
-        raise TypeError("first argument must be string or compiled pattern")
-    if flags & T:
-        import warnings
-        warnings.warn("The re.TEMPLATE/re.T flag is deprecated "
-                  "as it is an undocumented flag "
-                  "without an obvious purpose. "
-                  "Don't use it.",
-                  DeprecationWarning)
-    p = _compiler.compile(pattern, flags)
-    if not (flags & DEBUG):
+
+    key = (type(pattern), pattern, flags)
+    # Item in _cache should be moved to the end if found.
+    p = _cache.pop(key, None)
+    if p is None:
+        if isinstance(pattern, Pattern):
+            if flags:
+                raise ValueError(
+                    "cannot process flags argument with a compiled pattern")
+            return pattern
+        if not _compiler.isstring(pattern):
+            raise TypeError("first argument must be string or compiled pattern")
+        if flags & T:
+            import warnings
+            warnings.warn("The re.TEMPLATE/re.T flag is deprecated "
+                    "as it is an undocumented flag "
+                    "without an obvious purpose. "
+                    "Don't use it.",
+                    DeprecationWarning)
+        p = _compiler.compile(pattern, flags)
+        if flags & DEBUG:
+            return p
         if len(_cache) >= _MAXCACHE:
-            # Drop the oldest item
+            # Drop the least recently used item.
+            # next(iter(_cache)) is known to have linear amortized time,
+            # but it is used here to avoid a dependency from using OrderedDict.
+            # For the small _MAXCACHE value it doesn't make much of a difference.
             try:
                 del _cache[next(iter(_cache))]
             except (StopIteration, RuntimeError, KeyError):
                 pass
-        _cache[type(pattern), pattern, flags] = p
+    # Append to the end.
+    _cache[key] = p
+
+    if len(_cache2) >= _MAXCACHE2:
+        # Drop the oldest item.
+        try:
+            del _cache2[next(iter(_cache2))]
+        except (StopIteration, RuntimeError, KeyError):
+            pass
+    _cache2[key] = p
     return p
 
 @functools.lru_cache(_MAXCACHE)

Lib/test/test_asyncio/test_selector_events.py

@@ -61,8 +61,10 @@ def setUp(self):
     def test_make_socket_transport(self):
         m = mock.Mock()
         self.loop.add_reader = mock.Mock()
+        self.loop._ensure_fd_no_transport = mock.Mock()
         transport = self.loop._make_socket_transport(m, asyncio.Protocol())
         self.assertIsInstance(transport, _SelectorSocketTransport)
+        self.assertEqual(self.loop._ensure_fd_no_transport.call_count, 1)
 
         # Calling repr() must not fail when the event loop is closed
         self.loop.close()
@@ -78,8 +80,10 @@ def test_make_ssl_transport_without_ssl_error(self):
         self.loop.add_writer = mock.Mock()
         self.loop.remove_reader = mock.Mock()
         self.loop.remove_writer = mock.Mock()
+        self.loop._ensure_fd_no_transport = mock.Mock()
         with self.assertRaises(RuntimeError):
             self.loop._make_ssl_transport(m, m, m, m)
+        self.assertEqual(self.loop._ensure_fd_no_transport.call_count, 1)
 
     def test_close(self):
         class EventLoop(BaseSelectorEventLoop):

Lib/test/test_clinic.py

@@ -153,7 +153,7 @@ def test_right_only(self):
     def test_have_left_options_but_required_is_empty(self):
         def fn():
             clinic.permute_optional_groups(['a'], [], [])
-        self.assertRaises(AssertionError, fn)
+        self.assertRaises(ValueError, fn)
 
 
 class ClinicLinearFormatTest(TestCase):

Misc/NEWS.d/next/Core and Builtins/2022-10-06-23-13-34.gh-issue-97997.JQaJKF.rst

@@ -0,0 +1 @@
+Add running column offset to the tokenizer state to avoid calculating AST column information with pointer arithmetic.

Misc/NEWS.d/next/Library/2022-08-27-23-16-09.gh-issue-96346.jJX14I.rst

@@ -0,0 +1 @@
+Use double caching for compiled RE patterns.

Misc/NEWS.d/next/Library/2022-08-29-12-35-28.gh-issue-96073.WaGstf.rst

@@ -1 +1 @@
-In :mod:`inspect`, fix overeager replacement of "`typing.`" in formatting annotations.
+In :mod:`inspect`, fix overeager replacement of "``typing.``" in formatting annotations.