Skip to content

Race between dict_dealloc and split_keys_entry_added under free threading #130547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
hawkinsp opened this issue Feb 25, 2025 · 2 comments
Closed

Race between dict_dealloc and split_keys_entry_added under free threading #130547

hawkinsp opened this issue Feb 25, 2025 · 2 comments
Assignees
@corona10
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading type-bug An unexpected behavior, bug, or error

Comments

@hawkinsp
Copy link
Contributor

hawkinsp commented Feb 25, 2025 •
edited by bedevere-app bot
Loading

Bug report

Bug description:

In the following code it appears we race on the dk_nentries of a split keys dict.

import concurrent.futures
import functools
import threading
import _testcapi

num_threads = 32

def closure(b, i):
  b.wait()
  h = _testcapi.HeapCTypeWithManagedDict()
  h.__dict__[f"asdf{i}"] = "bar"

with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
  for i in range(1000):
    b = threading.Barrier(num_threads)
    for i in range(num_threads):
      executor.submit(functools.partial(closure, b, i))

TSAN report:

WARNING: ThreadSanitizer: data race (pid=3466735)
  Read of size 8 at 0x7ffa06cba218 by thread T32:
    #0 dict_dealloc /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:3170:42 (python3.13+0x26b02f) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #1 _Py_Dealloc /usr/local/google/home/phawkins/p/cpython/Objects/object.c:2935:5 (python3.13+0x2906a2) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #2 _Py_MergeZeroLocalRefcount /usr/local/google/home/phawkins/p/cpython/Objects/object.c (python3.13+0x2906a2)
    #3 Py_DECREF /usr/local/google/home/phawkins/p/cpython/./Include/object.h:913:13 (python3.13+0x271cf7) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #4 Py_XDECREF /usr/local/google/home/phawkins/p/cpython/./Include/object.h:1042:9 (python3.13+0x271cf7)
    #5 _PyObject_SetManagedDict /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:7151:13 (python3.13+0x271cf7)
    #6 PyObject_ClearManagedDict /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:7175:9 (python3.13+0x272317) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #7 heapmanaged_dealloc /usr/local/google/home/phawkins/p/cpython/./Modules/_testcapi/heaptype.c:822:5 (_testcapi.cpython-313t-x86_64-linux-gnu.so+0x21566) (BuildId: bcb92e2e6763ccff2c40c066ebaf418e7c3dee07)
    #8 _Py_Dealloc /usr/local/google/home/phawkins/p/cpython/Objects/object.c:2935:5 (python3.13+0x2906a2) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #9 _Py_MergeZeroLocalRefcount /usr/local/google/home/phawkins/p/cpython/Objects/object.c (python3.13+0x2906a2)
    #10 Py_DECREF /usr/local/google/home/phawkins/p/cpython/./Include/object.h:913:13 (python3.13+0x227b98) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #11 frame_dealloc /usr/local/google/home/phawkins/p/cpython/Objects/frameobject.c:1728:13 (python3.13+0x227b98)
    #12 _Py_Dealloc /usr/local/google/home/phawkins/p/cpython/Objects/object.c:2935:5 (python3.13+0x2906a2) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #13 _Py_MergeZeroLocalRefcount /usr/local/google/home/phawkins/p/cpython/Objects/object.c (python3.13+0x2906a2)
    #14 Py_DECREF /usr/local/google/home/phawkins/p/cpython/./Include/object.h:913:13 (python3.13+0x4bde23) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #15 Py_XDECREF /usr/local/google/home/phawkins/p/cpython/./Include/object.h:1042:9 (python3.13+0x4bde23)
    #16 tb_dealloc /usr/local/google/home/phawkins/p/cpython/Python/traceback.c:188:5 (python3.13+0x4bde23)
    #17 _Py_Dealloc /usr/local/google/home/phawkins/p/cpython/Objects/object.c:2935:5 (python3.13+0x2906a2) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #18 _Py_MergeZeroLocalRefcount /usr/local/google/home/phawkins/p/cpython/Objects/object.c (python3.13+0x2906a2)
    #19 Py_DECREF /usr/local/google/home/phawkins/p/cpython/./Include/object.h:913:13 (python3.13+0x4bdda3) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #20 Py_XDECREF /usr/local/google/home/phawkins/p/cpython/./Include/object.h:1042:9 (python3.13+0x4bdda3)
    #21 tb_dealloc /usr/local/google/home/phawkins/p/cpython/Python/traceback.c:187:5 (python3.13+0x4bdda3)
    #22 _Py_Dealloc /usr/local/google/home/phawkins/p/cpython/Objects/object.c:2935:5 (python3.13+0x2906a2) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #23 _Py_MergeZeroLocalRefcount /usr/local/google/home/phawkins/p/cpython/Objects/object.c (python3.13+0x2906a2)
    #24 Py_DECREF /usr/local/google/home/phawkins/p/cpython/./Include/object.h:913:13 (python3.13+0x209a86) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #25 BaseException_clear /usr/local/google/home/phawkins/p/cpython/Objects/exceptions.c:87:5 (python3.13+0x209a86)
    #26 NameError_clear /usr/local/google/home/phawkins/p/cpython/Objects/exceptions.c:2228:12 (python3.13+0x210fac) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #27 NameError_dealloc /usr/local/google/home/phawkins/p/cpython/Objects/exceptions.c:2235:5 (python3.13+0x210fac)
    #28 _Py_Dealloc /usr/local/google/home/phawkins/p/cpython/Objects/object.c:2935:5 (python3.13+0x2904a5) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #29 merge_queued_objects /usr/local/google/home/phawkins/p/cpython/Python/brc.c:110:13 (python3.13+0x3dca2a) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #30 _Py_brc_merge_refcounts /usr/local/google/home/phawkins/p/cpython/Python/brc.c:131:5 (python3.13+0x3dca2a)
    #31 _Py_HandlePending /usr/local/google/home/phawkins/p/cpython/Python/ceval_gil.c:1289:9 (python3.13+0x45358e) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #32 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:5160:17 (python3.13+0x3f2f00) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #33 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #34 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #35 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #36 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef2d5) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #37 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:62:18 (python3.13+0x1ef2d5)
    #38 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ead4a) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #39 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1ead4a)
    #40 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e264b) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #41 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #42 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #43 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #44 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef38f) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #45 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef38f)
    #46 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #47 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #48 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #49 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x5649a2) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #50 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdca7) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)

  Previous atomic write of size 8 at 0x7ffa06cba218 by thread T2:
    #0 _Py_atomic_store_ssize_relaxed /usr/local/google/home/phawkins/p/cpython/./Include/cpython/pyatomic_gcc.h:481:3 (python3.13+0x273284) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #1 split_keys_entry_added /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:225:5 (python3.13+0x273284)
    #2 insert_split_key /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:1743:9 (python3.13+0x273284)
    #3 insertdict /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:1796:25 (python3.13+0x263496) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #4 setitem_take2_lock_held /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:2507:12 (python3.13+0x262996) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #5 _PyDict_SetItem_Take2 /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:2515:11 (python3.13+0x262bcf) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #6 PyDict_SetItem /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:2535:12 (python3.13+0x262bcf)
    #7 dict_ass_sub /usr/local/google/home/phawkins/p/cpython/Objects/dictobject.c:3340:16 (python3.13+0x27557a) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #8 PyObject_SetItem /usr/local/google/home/phawkins/p/cpython/Objects/abstract.c:232:19 (python3.13+0x1b9488) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #9 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:5777:27 (python3.13+0x3f5d0b) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #10 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #11 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #12 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #13 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x5722c2) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #14 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x5722c2)
    #15 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #16 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #17 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #18 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4832) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #19 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #20 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #21 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #22 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef38f) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #23 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef38f)
    #24 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #25 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #26 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #27 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x5649a2) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)
    #28 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdca7) (BuildId: eba64ecb5d6238e288495b7377dad10e1b152da3)

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs
@hawkinsp hawkinsp added the type-bug An unexpected behavior, bug, or error label Feb 25, 2025
@hawkinsp hawkinsp mentioned this issue Feb 25, 2025
Closed
@picnixz picnixz added topic-free-threading interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Feb 25, 2025
@corona10 corona10 self-assigned this Feb 25, 2025
@corona10
Copy link
Member

I will take a look :)

@bedevere-app bedevere-app bot mentioned this issue Mar 3, 2025
Merged
corona10 added a commit to corona10/cpython that referenced this issue Mar 3, 2025
@corona10
pythongh-130547: Fix race between dict_dealloc and split_keys_entry_a…
0b46f65 
…dded
corona10 added a commit that referenced this issue Mar 4, 2025
@corona10
gh-130547: Fix race between dict_dealloc and split_keys_entry_added (g…
80e6d3e 
…h-130778)
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Mar 4, 2025
@corona10 @miss-islington
pythongh-130547: Fix race between dict_dealloc and split_keys_entry_a…
a778b8c 
…dded (pythongh-130778)

(cherry picked from commit 80e6d3e)

Co-authored-by: Donghee Na <[email protected]>
@bedevere-app bedevere-app bot mentioned this issue Mar 4, 2025
Merged
corona10 added a commit that referenced this issue Mar 5, 2025
@miss-islington @corona10
[3.13] gh-130547: Fix race between dict_dealloc and split_keys_entry_…
ffef9b0 
…added (gh-130778) (gh-130833)

gh-130547: Fix race between dict_dealloc and split_keys_entry_added (gh-130778)
(cherry picked from commit 80e6d3e)

Co-authored-by: Donghee Na <[email protected]>
@corona10
Copy link
Member

corona10 commented Mar 5, 2025 •
edited
Loading

@hawkinsp Fixed, please let us know if this issue is reproduced again, thanks for all the reports from JAX!

@corona10 corona10 closed this as completed Mar 5, 2025
hawkinsp added a commit to hawkinsp/jax that referenced this issue Mar 5, 2025
@hawkinsp
Remove a TSAN suppression.
40e1a2a 
python/cpython#130547 has been marked as fixed and backported to 3.13, so this suppression should no longer be necessary.
@hawkinsp hawkinsp mentioned this issue Mar 5, 2025
Merged
seehwan pushed a commit to seehwan/cpython that referenced this issue Apr 16, 2025
@corona10 @seehwan80
pythongh-130547: Fix race between dict_dealloc and split_keys_entry_a…
b2e2c95 
…dded (pythongh-130778)
@vfdev-5 vfdev-5 mentioned this issue Apr 16, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@corona10 corona10

Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hawkinsp @corona10 @picnixz