Support multiple workers / instances behind load balancer / reverse proxy (for example Gunicorn or similar)

[aersam]

This is a proposal after having discussed an issue in #1193

Currently nicegui cannot be used behind gunicorn or another reverse proxy without affinity support. This may lead to confusion and also I think it's straightforward to remove that limitation.

Reverse Proxies are quite common with FastAPI/Python since otherwise the Python GIL limits you quite a bit and a single long running request can block others (if not using async/await, sometimes you can't)

I think it would be possible for nicegui to implement support for running behind a reverse proxy without too much effort, but maybe I'm wrong. This is because once the websocket connection is established, only a single worker is involved anyway.
One would need persist the client(id/data) to disk on initial page render and then read it from there if needed. After the websocket handshake, everything works with the websocket which implies staying on the same worker. So only one hand-over between workers/processes is required.

Do you think that's something to consider? Or do I simply things too much?

[rodja]

Thanks for bringing this up @aersam. We would love to remove this limitation. But it might not be so simple.

One would need persist the client(id/data) to disk on initial page render and then read it from there if needed.

The data part may become tricky. Let us consider a simple scenario with a load balancer / reverse proxy in the front and two workers / instances named A and B in the back. The correct page-builder function (decorated with @ui.page) is executed when requesting the initial page. For example on worker A. While execution, arbitrary state can be created on the worker. For example opening an usb camara device or creating a database connection. After loading the html response the client establishes a websocket connection but may be routed to worker B by the load balancer. I do not know any way to transfer the state from A to B.

Last week @falkoschindler voiced a possible alternative approach: what if the page builder does not execute the decorated function directly but only serves a super tiny, generic html page which establishes a websocket connection. Then the the page builder is executed after the websocket connection is established. We have not found the time to investigate this approach. Anybody interested in giving it a shot?

[aersam]

I am unsure how much time I find for it, but will do this sooner or later as we will need it

[rodja]

FYI: @evnchn just tried this in #4487.

[rodja]

If we're running the page builder on incoming websocket instead of incoming http request, we need to abandon the ability to return Starlette/FastAPI responses. And we won't have access to the request parameters any longer.

Another problem with multiple workers is the loss of global state. For example in a simple chat app, only the users on the same worker would see each other.

[aersam]

I think people are aware of the loss of global state if working with multiple workers, that's the same thing in every other framework, too.

Access to request parameter and not being able to return Responses is a big thing to me, however. It's the only meaningful place to do authentication, so I guess we need a different solution.

What if one could add a parameter sideeffect=False to ui.page which would allow to transfer the state between worker on init? Since the Id is strictly increment only, I guess it should work to just reexecute it on another worker. The first worker would somehow need to be able to remove it's state after some timeout since he won't receive the connection

Or allow something more involved which allows splitting the logic in two handlers:

@ui.page('/documentation')
def documentation_page(page) -> None:
    @page.pre_render(req: Request) ## no access to events/client. used for SEO/access to parameters etc
    def pre_render():
        ui.html("Loading...") # html would be removed on render
        if not req.bla
            return Response() # if a response object is returned, no connection wanted
        return None # maybe allow a json-dumpable dict that is passed to render?, even on other worker

    @page.render()
    def render():
        add_head_html()
        menu = side_menu()
        add_header(menu)
        ui.add_head_html('<style>html {scroll-behavior: auto;}</style>')
        with ui.column().classes('w-full p-8 lg:p-16 max-w-[1250px] mx-auto'):
            section_heading('Reference, Demos and more', '*NiceGUI* Documentation')
            documentation.create_full()

But looks too complicated... Would need to be optional to use

[rodja]

If we could auto-discover and reach all other workers we may be able to route the websocket stream to the correct worker.

Or we simply write a tutorial on how to use a load balancer like Traefik which supports session affinity instead of plain gunicorn.

[rodja]

I'm not too confident you never run into a blocking problem in a larger production app.

Sure, it's not easy. But I think it's still the best option for running a larger production app. In our robotics framework RoSys which uses NiceGUI as a basis we can activate asyncio warnings for tasks which block to long: https://github.com/zauberzeug/rosys/blob/main/rosys/analysis/asyncio_warnings.py

Also because many libraries just don't support async/await, just imagine some people doing some unwise pandas stuff.

Pandas operations are mostly not I/O bound but cpu bound and need to be executed in a separate process (see run.cpu_bound in RoSys).

For scenarios like these, gunicorn is a big help

I disagree. Gunicorn covers up the inherent problem of having blocked the event loop. It will still affect everything running on the specific worker. Hence a portion of your users will still experience the lag.

[v479038280]

If we can learn from streamlit about how to handle such delay?

[aersam]

@rodja: Well you're right in that's the proper way of doing things. I just have a quite a special group of people doing python that's not mainly devs but more coming from a data background without too much knowledge on resource management. Gunicorn is a bit of a duplicate assurance to me that no stupid thing like a memory leak or so blows up everything. If I have 4 workers and one is blown up, gunicorn will restart it and it will save me lot's of tickets, but not all. Of course it's not a solution, it's a workaround.

However I must admit that I did not see how complex this will be and therefore it's maybe just simpler to use a real load balancer with affinity support. What you should be doing however is to have a page to direct users to in case they experience the handshake error, to have some guidance there

[rodja]

Thanks for explaining @aersam. I totally understand the issue but am not sure how to fix it. I've created a new entry in our FAQs: https://github.com/zauberzeug/nicegui/wiki/FAQs#how-to-avoid-the-reloading-because-handshake-failed-javascript-error-message-which-results-in-a-full-page-reload

[rodja]

We have recently released Redis storage support. With that you can sync state between multiple instance. The example code also demonstrates how to setup sticky sessions with Traefik:

https://github.com/zauberzeug/nicegui/blob/bb5edf45424846f76ffee4406931a6f97b349a80/examples/redis_storage/docker-compose.yml#L13C7-L13C70

[Stefan2016]

Thanks for explaining @aersam. I totally understand the issue but am not sure how to fix it. I've created a new entry in our FAQs: https://github.com/zauberzeug/nicegui/wiki/FAQs#how-to-avoid-the-reloading-because-handshake-failed-javascript-error-message-which-results-in-a-full-page-reload

Do you have experience setting up session affinity with ngnix and nicegui? Would be interesting to see an example.

[rodja]

This feature request is more about supporting multiple workers for proxies which do not support session affinity. But nginx does. See #3038 for a discussion about such a setup.

[barrywhart]

Has anyone considered the possibility of storing the Client instances in a data store that is shared across workers / instances? In the case of multiple gunicorn workers, the Client instances could be stored in memory in a separate singleton process or in a data store such as shelve or SQLite.

The clients would be serialized using pickle, dill, cloudpickle. (Note that dill and cloudpickle have better support than pickle for serializing code such as Python lambdas.)

For scaling across multiple servers, the Client instances could be stored in Redis or a traditional database.

I understand the earlier point about side effects, but I think this can be minimized or avoided through documentation. The basic guidance would be to that state is only accessible via fields of the Element objects that make up the page. For my use case, the application is a pretty traditional web application that ultimately triggers some long-running automation jobs (e.g. applying Terraform changes).

[rodja]

As I wrote in #1539 (comment):

While execution, arbitrary state can be created on the worker. For example opening an usb camara device or creating a database connection.

So you do not only need to sync the client instances but everything. Mabye in very defined special scenarios it may work... but I think it its nothing we could support from the general NiceGUI library side.

[barrywhart]

very defined special scenarios it may work

I think this (well-defined state) is the typical case for many applications. How many web applications have a USB camera device connected to the server?

Regardless, perhaps the way to approach this would be to enhance NiceGUI to support pluggable state backends. Then people could select or build a state backend based on their needs. Another advantage of this approach is that the best backend could vary based on the scale of the application. E.g. an application with a handful of users may choose to use a SQL backend, while a high-volume application might use Redis or a NoSQL backend.

[barrywhart]

Related: Last year, I contributed a pluggable backend for MLflow (PR link). In that case, it was for authentication, not for state management.

I mention this because, if there's potential interest in this idea, I may be able to contribute.

I think this capability could significantly expand the potential number of users for NiceGUI. It's a really great package, and I'd love it if it were suitable for higher-volume applications.

[T2XX]

can we using SSE technology to avoid websocket http upgrade? so that we can use gunicorn works.

[T2XX]

see example

[rodja]

SSE (Server-Sent Events) are unidirectional and hence are not suited for interactive applications as far as I know. Also they would have the same problem as websockts: the state for one client must be completely managed in one worker and can not be shared between processes.

[evnchn]

Stumbled upon this while searching some more.

Want to mention that if you go #4474 and add workers=10, or even workers=100 to uvicorn.Config, nothing blows up in errors, and page visits (at least for simple pages, no inteactivity no nothing) this seems to work fine.

Summarizing the discussion, with regards to increasing workers to uvicorn, the issues seem to be:

1. Can't transfer state between workers, so you better hope the worker that served you continues to serve you again, or lose state.
2. WebSocket may go to wrong worker, worker don't know how to respond to foreign requests
3. No more global state (think: auto-index page is the biggest loser)
4. No worker affinity for Gunicorn and Uvicorn?

Under the context of NiceGUI 2.10.0, if all workers do not matter their state, as they consult the Redis storage, can it then work?

[rodja]

Unfortunately, your second point ("WebSocket may go to wrong worker") is the most problematic and not solvable through usage of Redis storage and banning all global state: a page is requested via http and NiceGUI builds up state for this (eg. client, context,...) and waits for the websocket handshake. Other workers would not know about that state.

[evnchn]

Hmm, that's the toughest pill to swallow right now.

So, I thought about it for a long time. It seems that both Gunicorn, Uvicorn, and Hypercorn does not have the worker affinity, and the problem of "first talking to worker A, and then getting transfer to worker B who has no idea what is going on" is real.

But, I thought about it. WebSocket is basically a sugarcoated TCP tunnel, right? Guess what else is a socket (or at least somewhat like one). That's right, QUIC, the star of the show in HTTP/3. Think #184 (reply in thread).

Gunicorn and Uvicorn may do us no good, but based strictly on how Hypercorn works, if we then switch the transport of socket.io from WebSocket to polling on the HTTP straight away, then we may "trick Hypercorn into not releasing its worker from the client"? Because how else could it work. Multiple workers responding to the same QUIC tunnel?

And that could also have the added benefit that since QUIC is UDP-based when WebSocket is TCP-based, that it becomes faster in general.

[rodja]

Sounds interesting. But difficult. As #4476 shows. There you write

HTTP3, connection refused with Chrome, Edge, Supermium, and curl bin for Windows

[evnchn]

I think with the fact that if it is HTTP/2 the WebSocket request gets downgraded to HTTP1.1, serving a separate WebSocket connection over HTTP3 is not possible.

But I want to try swapping out the transport of socket.io to polling and run on HTTP3, if I do get HTTP3 to work... (Or I can try it with HTTP2 too I guess, but then it'll be slower than WebSocket, with possible speed gains in initial connection only)

If it doesn't work, I don't think it's worth the pursuit...

[evnchn]

HTTP3 doesn't work. WebTransport (QUIC version of WebSocket) not supported by Python-SocketIO. All dreams are shattered, and let's stick to HTTP1.1 for the time being.

Only advantage for hypercorn, as it stands, would be support for Trio, which we can make work if we switch over to AnyIO.

Or, HTTP2 may be slightly better for initial page load over really terrible networks, which I have to contend with (and I hope you don't), so I may consider to switch to hypercorn HTTP2 for my own project.

[evnchn]

Let me also think:

• What if the page builder does not execute the decorated function directly but only serves a super tiny, generic html page which establishes a websocket connection.
• Then the the page builder is executed after the websocket connection is established, but we need to abandon the ability to return Starlette/FastAPI responses. And we won't have access to the request parameters any longer.

Is that so? From working with await ui.context.client.connected() I already learnt that it is possible to essentially "cut an async function into 2", with the first part content served over HTTP and anything after that WebSocket.

I am thinking

def get_authenticated_ids():
    # you would read the ids in a manner that it is possible to add the ids and it works for all workers
    # Write to a JSON file? Redis/memcached? 
     return []

@ui.page("/)
async def serve_super_tiny_generic_html_then_render():
    # Here, you put stuff which is almost like "reflex response"
    # in a sense that everyone shouts when they stubbed their toe, all workers reject unauthenticated requests
    if app.storage.browser['id'] not in get_authenticated_ids():
        return RedirectResponse('/login')

    await ui.context.client.push_html()
    # afterwards, you forfeit any possibility to return RedirectResponse, going into a special mode where the content otherwise sent to the client via HTTP is sent via WebSocket

    random_string = "".join(random.choices("abcdefghijklmnopqrstuvwxyz", k=10)) # some stuff specific to this worker, other workers don't know
    ui.label(f"Random string: {random_string}")

    await ui.context.client.connected()
    # afterwards, we switch mode back to original operation

    # Here, we run the normal logic of continuously sending UI updates and JavaScript execution requests (and the alike) that we normally would with WebSocket
    # make the string capital and display it
    ui.label(f"Capitalized: {random_string.upper()}") # must be the same worker to possibly respond to this. 
    result = await ui.run_javascript('1+100') # expects 101

Then, you can look forward to await asyncio.wait([task,asyncio.create_task(client._waiting_for_connection.wait()),asyncio.create_task(client._waiting_for_push_html.wait())], timeout=self.response_timeout, return_when=asyncio.FIRST_COMPLETED) in the client.py, where we will act in 3 ways:

• The async page wishes to be multi-worker-compliant and agrees to forfeiting all global states, and pushes the tiny HTML.
• The async page just wants a normal HTML to be pushed and waiting for typical client connection.
• The async page actually simply returns, with no ui.context.client awaits, and we also return whatever was returned if that's the case.

A bit complicated, though 😅

[evnchn]

But why would I want to do both await ui.context.client.push_html() and also await ui.context.client.connected()

By me pushing a tiny HTML I am looking for a client to talk back to me already, and is ready if the worker who then follows up is another worker.

So maybe @ui.page(allow_worker_switchover=True), in a sense that if you do that, you allow that it is the case that another worker is the one who runs the code after that line.

def get_authenticated_ids():
    # you would read the ids in a manner that it is possible to add the ids and it works for all workers
    # Write to a JSON file? Redis/memcached? 
    return []

def get_username_from_id(id):
    return "evnchn"

@ui.page("/", allow_worker_switchover=True)
async def serve_super_tiny_generic_html_then_render():
    # Here, you put stuff which is almost like "reflex response"
    # in a sense that everyone shouts when they stubbed their toe, all workers reject unauthenticated requests
    if app.storage.browser['id'] not in get_authenticated_ids():
        return RedirectResponse('/login')

    ui.label("Random string from worker A" + "".join(random.choices("abcdefghijklmnopqrstuvwxyz", k=10)))
    # This would be fine though, but don't expect Worker B to be able to read it
    # If you want, you can use document.getElementById("c1").innerText if you want to force it, but Worker B gets to know Worker A's details via the browser only. 

    # username = get_username_from_id()
    # this is a BAD IDEA. Not here! Because otherwise Worker B has no idea!

    # Suppose Worker A was running all the code above and then also the following line
    await ui.context.client.connected()
    # The browser may talk WebSocket, talk to Worker B

    username = get_username_from_id(id) # Worker B
    ui.label('Welcome'+get_username_from_id(app.storage.browser['id'])
    
    # your typical page here, except the client is always connected

If allow_worker_switchover=True, we re-execute the @ui.page logic for Worker B, but regarding await ui.context.client.connected(), we skip every ui element construction before that line, and don't block either because it is connected.

[evnchn]

Basically:

Old approach: Unknown session = fail and reload.

New approach: Unknown session and allow_worker_switchover=True, we goto line_after("await ui.context.client.connected()") and resume execution

[evnchn]

After a night of sleep, let me summarise it a bit better.

We want to split it so that under allow_worker_switchover=True, only worker-agnostic code is placed before await ui.context.client.connected(), with worker-specific code placed below.

[rodja]

Creative idea! I'm not sure if the complications arising from it are worth the effort. But would you like to create a spike somehow to demonstrate feasibility?

[evnchn]

I think it could be doable. I'll have to read up the source code for a bit to see where the WebSocket is handled, and do a bit of "software trickery" here or there.

The code may be ugly and hacky at first, but I'm sure someone will clean up the code later after the feasibility check is done 😅

[rodja]

After going through this discussion again, here is a quick summary from my point of view:

• before you embark on multiple workers/instances you should properly use async/await throughout your code base (that is much easier than anything else which follows below)
• to run a single application on multiple instance, NiceGUI requires sticky sessions and some way to sync between the processes
• thanks to Redis storage, we now have an easy way to do the sync (alternatives like databases are also possible)
• Uvicorn/Gunicorn etc. use the operating system to route incoming traffic and do not support sticky sessions
• there are quite a lot of professional load balancers like Traefik, Nginx, HAProxy, proxy.py, ... which support sticky sessions but need special knowledge to set up
• a Traefik setup with Redis can be found in the examples
• in NiceGUI with multiple workers via custom request dispatcher with worker affinity #4483 a custom (and simple) load balancer is discussed right now

[evnchn]

Yes. But there are still cases that this multiple worker is useful:

• Adding nginx on top is quite some bulk and complexity.
• NiceGUI elements take some time to be created, adding up if there are many of them which you will see if you profile the code, and so does the element-to-JSON serialization process, which has quite an amount of nested calls.

About my proposal of @ui.page(allow_worker_switchover=True), there is also an added advantage that we can probably run 2 uvicorn instances, one which has 1 worker and handle all the pages without allow_worker_switchover, and the other one which has N workers and handle all the pages with allow_worker_switchover.

Then, we can possibly segregate between pages that are

• Simple but highly repetitive tasks where performance matters (think: the visitor's page, view-only, little interactivity)
• Complex but rarely accessed where a single worker following-through with the task matters (think: the admin's panel, many CRUD operations)

Also, I just like to see what heights NiceGUI can reach, always questioning what is possible and what is not.

I think I will try my hands on my proposed approach of @ui.page(allow_worker_switchover=True) #1539 (reply in thread), but only if I have time, though.

---

And, here is a brief guide so that we don't lead people astray if they simply want basic speed improvements

If you are facing speed issues

First take these steps which are "still under the normal operation zone of NiceGUI"

1. Check if all your own CPU bound tasks (computing 1000 digits of Pi and returning a string, say) and IO bound tasks (making a web request to some very slow server somewhere) are all using the desired way of running CPU bound tasks and IO bound tasks
2. Check if the number of NiceGUI elements being served is minimal as it should be, employ techniques such as server-side pagination, or potentiall exotic techniques if need be such as grouping things into advanced custom elements

If you are very sure step (1) and (2) are well done, and you still see NiceGUI occupying 1 CPU core and pinning that core to 100%, then you know, hey, it is NiceGUI becoming CPU bound (this is decidedly rare. Maybe you have no choice but to render 10000 elements all on screen, and that takes a solid 1 second of 100% CPU time). You can choose to let it be, or take these following steps which are "more advanced and outside of what 'normal' people would do"

3. As NiceGUI does not work with multiple Uvicorn workers as of now, until something changes in Support multiple workers / instances behind load balancer / reverse proxy (for example Gunicorn or similar) #1539, you have no choice but to use NiceGUI across multiple independent processes to overcome the GIL, accessible over different internal ports. This means you must give up shared storage which is provided to you because it is in a single worker. No writing to a global variable and then getting it back later. That simply doesn't happen
   a. More clearly about NiceGUI side, you see https://nicegui.io/documentation/storage, anything that is server-side is gone. The only useful one left is app.storage.browser['id'], which you must use to interact with some database elsewhere,
   b. Or, use Redis storage for app.storage
4. Then, you need some solution for distributing the requests to the N workers who can then use your N CPU cores. A traefik example was hiding in the Redis example, and here are the outlines of what your proxy solution must be able to do, which is to redirect requests based on the ID in the session JWT in the cookie. If it does sticky cookie that would be even better, since the JWT actually changes from time to time because it was re-issued with a different validity end date on every page load.

[rodja]

Thanks for the great summary on what can be done to improve performance. Based you your text I created a new FAQ: https://github.com/zauberzeug/nicegui/wiki/FAQs#what-can-i-do-to-improve-performance