Potential reference counting error in Python module

[ksun212]

System Information

OpenCV python version: 4.6.0.66
Operating System / Platform: Ubuntu 20.04
Python version: 3.9.15

Detailed description

Hello! I have a Cpython version 3.9.15 built with NDEBUG flag. When I import cv2 in it, it aborted with garbage collection error.

Modules/gcmodule.c:116: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x55c596b9ad90
object refcount : 8
object type     : 0x55c596331780
object type name: type
object repr     : <class 'cv2.utils.nested.ExportClassName'>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x0000149adc163600 (most recent call first):
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1173 in create_module
  File "<frozen importlib._bootstrap>", line 565 in module_from_spec
  File "<frozen importlib._bootstrap>", line 666 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
  File "/home/user/real_pure_python/cpython-3.9/Lib/importlib/__init__.py", line 127 in import_module
  File "/home/user/real_pure_python/cpython-3.9/my_python/lib/python3.9/site-packages/cv2/__init__.py", line 153 in bootstrap
  File "/home/user/real_pure_python/cpython-3.9/my_python/lib/python3.9/site-packages/cv2/__init__.py", line 181 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 850 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<stdin>", line 1 in <module>
Aborted (core dumped)

I think about this for some time and guess the problem is that the class "ExportClassName" is added to two modules in "registerNewType", which calls "PyModule_AddObject", which decrease the reference count of "ExportClassName" by one in the end. Thus, before calling PyModule_AddObject, one should either hold an existing reference of "ExportClassName" or increase the reference by one.

Since I am new to opencv, there may be something I misunderstand. Thanks in advance.

Steps to reproduce

To reproduce this bug, import cv2 in any Cpython built with NDEBUG flag should be enough.

import cv2

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

[spikethehobbitmage]

This error message means that the garbage collector found more references to an object than are recorded in the object's refcount. This can be caused by improper reference counting or by exposing borrowed references to the garbage collector. See python/cpython#80142 for an example. In that case it was fixed by converting borrowed references into owned references.

Calling the garbage collector immediately before loading cv2 allows it to load, but the interpreter will always crash the next time the garbage collector runs.

The error isn't limited to cv2.utils.nested.ExportClassName, but can also occur with cv2.utils.ClassWithKeywordProperties. It seems likely that other objects are affected as well.

I don't know python or opencv internals well enough for it to be immediately obvious where the problem is, but I can confirm that the problem occurs somewhere in the init_body function of opencv/modules/python/src2/cv2.cpp.

Edit: Add full paths for affected objects.

Edit2: ksun212's intuition appears to be correct. cv2.cpp::registerNewType calls cv2.cpp::registerTypeInModuleScope twice, but cv2.cpp::registerTypeInModuleScope calls PyModule_AddObject, which steals the reference. cv2.cpp::init_body is also missing a Py_DECREF(opencv_error).

[spikethehobbitmage]

I'm putting together a PR to fix this. It's pretty simple, just changing three lines of code.

1. Fix registerTypeInModuleScope so registerNewType doesn't double-steal sometimes.
2. Make CVPY_TYPE_INIT_DYNAMIC clean up after itself.

CVPY_TYPE_INIT_STATIC isn't changed because python can't free static objects, so it's reference shouldn't be cleaned.