Leaked Semaphore Objects

[rodja]

Discussed in #4129

^{Originally posted by ChipsAHoai December 20, 2024}

Question

I am having leaked semaphore issues when running nicegui with fastapi. Initially, I thought it was my code, however, I just tested this on Zauberzeug's repo running the start.sh bash script under examples/fastapi and immediately closed down the uvicorn server and it still shows leaked semaphores. Is there a solution for this?

Reproduction:

cd examples/fastapi
./start.sh dev
Starting Uvicorn server in development mode...
INFO:     Will watch for changes in these directories: ['nicegui/examples/fastapi']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [54107] using WatchFiles
INFO:     Started server process [54109]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [54109]
INFO:     Stopping reloader process [54107]
/opt/homebrew/Cellar/[email protected]/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown

^CINFO: Shutting down is the line where I press CTRL-C to abort the execution.

It also happens when starting native mode and reloading:

from nicegui import ui

ui.label("hello")

ui.run(native=True)

Pressing CTRL-C yields also a leaked semaphore objects warning.

[rodja]

@falkoschindler do you remember why 942e5e4 reverted the fix #4132?

[falkoschindler]

@rodja See #4244.

[devturner]

On version nicegui==2.11.1
I have been seeing this pretty often with my use of the app, I have seen it jump as high as 60 objects! it does not seem to have any noticeable affect yet.

resource_tracker.py:276: UserWarning: resource_tracker: There appear to be 14 leaked semaphore objects to clean up at shutdown

[TomFreudenberg]

For me it is the same and I synthesized just to the import

import nicegui

import time
import sys
import os

print('SLEEP FOR STOP')
time.sleep(3)

print('RESTART')
python = sys.executable
os.execv(python, ['python'] + sys.argv)

If you comment out the import nicegui all works fine
If you leave the importas is:

.pyenv/versions/3.12.7/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 11 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Just run by python mytest.py

[TomFreudenberg]

It seems to be an issue while using multiprocessing Queue

This is eneough to get leaked memory:

import time
import sys
import os

from multiprocessing import Queue
q = Queue()

print('SLEEP FOR STOP')
time.sleep(1)

print('RESTART')
python = sys.executable
os.execv(python, ['python'] + sys.argv)

if change the line to

q = None

no issue :-(

[TomFreudenberg]

If someone is interested in going deeper, I realized that the issue raises when Lockor Semaphore message gets written to pipe (file descriptor)

file: multiprocessing/resource_tracker.py [183]

    def _send(self, cmd, name, rtype):
        ...
        msg = '{0}:{1}:{2}\n'.format(cmd, name, rtype).encode('ascii')
        ...
        nbytes = os.write(self._fd, msg)

If I drop the message like:

        msg = b''

no leak exist.

self._fd is pointing to

file: multiprocessing/util.py [451]

_posixsubprocess.fork_exec(
    args, [path], True, passfds, None, None,
    -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
    False, False, -1, None, None, None, -1, None,
    subprocess._USE_VFORK)

for args:

[b'.venv/bin/python', '-c', 'from multiprocessing.resource_tracker import main;main(7)']

and path:

b'.venv/bin/python'

---

I am running on OSX with python 3.12.7

Not sure how to proceed here ?

@falkoschindler - any suggestions ?

[TomFreudenberg]

GOTCHA :-)

If you forward the process into a new one by os.execv and using multiprocessing Queues, you have to make sure that the cleanup is run, which is normally run on catching sys.exit by multiprocessing but not on execv.

The issue is solved by this:

import time
import sys
import os

import multiprocessing

q = multiprocessing.Queue()

print('SLEEP FOR STOP')
time.sleep(1)

print('RESTART')

# important cleanup locks and semaphores
multiprocessing.util._exit_function()

python = sys.executable
os.execv(python, ['python'] + sys.argv)

---

not sure if this may help some others here but for tokeo it sole my reload for running nicegui.

Kindly
Tom

[falkoschindler]

Looks promising, @TomFreudenberg!
I wonder how we can integrate this approach into NiceGUI. Would you (or anyone from the community) like to create a pull request? 🙂

[TomFreudenberg]

Hi @falkoschindler I will try a PR within the next couple of time.
Are there more places in NiceGui where execv is used other than on reload for file changes?

[falkoschindler]

@TomFreudenberg Great!

Are there more places in NiceGui where execv is used other than on reload for file changes?

Not that I'm aware of.

[rodja]

@TomFreudenberg with c516d03 I put the fix into #4641 which needed it desperately. Due to changes in shutdown mechanics the "leaked semaphores" warning appeared in every app.

[TomFreudenberg]

Super @rodja, so I press my thumps that the PR is available soon.

Thanks for fixing this as well.

[TomFreudenberg]

@falkoschindler I saw that you reverted the changes. Let me know if more effort is necessary afterwards.

[falkoschindler]

@TomFreudenberg Oh yes, we noticed that #4641 doesn't seem to introduce new semaphore problems, but the same warnings already happen on main. So there is no need to solve this problem in this very PR. And as it turned out, fixing the semaphore objects still needs some work (tests are not passing yet). So @rodja suggested to extract this topic from the PR. He can certainly give some more information about what needs to be done.