Support for Bull / BullMQ libraries

[manast]

Due to the increasing popularity of Upstash as a Redis hosting solution, many Bull/BullMQ users are trying to deploy their queues and getting all kinds of weird behaviors since, at the moment, Upstash is not compatible with any of these two libraries.

The reason is not working is because Bull/BullMQ requires to create keys inside the Lua scripts, but Upstash requires the keys to be explicitly known before the scripts are executed.

In order to resolve this issue, it would probably be needed to make changes both in Bull/BullMQ as well as in Upstash.

[enesakar]

hey @manast . Do you have anything you plan on your side? How possible is it to call the lua scripts with predefined keys? @mdogan knows better but what I understood is this is the only thing blocks us.

[manast]

@enesakar I am waiting for the dev team to come back to me so that we can discuss potential solutions.

[pellizzetti]

Ran into this issue today, did they managed to contact you @manast? Any updates @mdogan?

[manast]

I have been contacted and offered all my help to solve this issue, and it seems they looked a bit into it but I do not know the status or their plan.

[ChristianIvicevic]

I am curious since I just tested Bull agains Upstash via Nest.js myself and it seems to work. The only caveat being that there are so many commands that I'd go broke, but it seems to work regardless.

[chronark]

It works as long as all keys remain in memory.
But we evict keys from memory after some time (they are of course durably stored on disk).

The problem is that bullmq is building dynamic keys inside of lua Scripts, which is highly discouraged by the official redis specs btw.

When executing a script on upstash, we first load all specified keys from disk to memory, but that's only possible for the ones that are explicitly provided as script arguments.

[chronark]

Important: to ensure the correct execution of scripts, both in standalone and clustered deployments, all names of keys that a script accesses must be explicitly provided as input key arguments. The script should only access keys whose names are given as input arguments. Scripts should never access keys with programmatically-generated names or based on the contents of data structures stored in the database.

from https://redis.io/docs/manual/programmability/eval-intro/#getting-started

[manast]

The problem is that bullmq is building dynamic keys inside of lua Scripts, which is highly discouraged by the official redis specs btw.

It has not been highly discouraged previously, although they have rephrased the recommendation recently. The reason for this recommendation is because Redis cannot fully take advantage of clustered setups, although there are workarounds for this https://docs.bullmq.io/bull/patterns/redis-cluster

Bull/BullMQ works really well on all Redis hosting solutions (Elasticache, GCP, Redis cloud, Redis Green, etc), but does not work on Upstash because it is a custom implementation that do not work exactly the same as the rest.

I have offered resources to try to find a solution but unfortunatelly this has not received enough priority from Upstash (or they think it is too difficult to solve) so it has not yet been resolved.

[manast]

For the record, I challenge anybody if they can find a bug in Bull/BullMQ related to using keys generated in LUA scripts when using any official Redis instance.

[ctrlaltdylan]

I'm a BullMQ user and a Upstash customer. I would transfer my hosting to Upstash if it was compatible with BullMQ. But this incompatibility has forced me to manage my background jobs on AWS.

[chronark]

@manast you are right it works with the official redis server implementation, because redis keeps everything in memory and our implementation moves entries to disk to allow us to provide the service at much lower cost than traditional redis vendors.

When implementing it, we were sticking with the guidelines and specs at redis.io and at this point the only solution would be to enable the option to keep everything in memory
That would make things way more expensive and commercially unviable to be used with a queue library like bullmq or others. They tend to be very noice and are not well suited for a per-request-pricing-model

Unfortunately making bullmq compatible is not as easy as fixing a few bugs.

@ctrlaltdylan
If you want to use a managed queue on Upstash, please try out QStash.
It offers many of the features of bullmq and some unique ones such as being able to run with serverless functions and more.

[manast]

Unfortunately making Bullmq compatible is not as easy as fixing a few bugs.

We do not know actually, because we never discussed the possibility in detail...

[manast]

Closing to signal that no further actions are expected in this regard.

[enesakar]

@manast We have worked on a solution and include in the next release if it passes our perf tests.

[manast]

Impressive!

[ianjkaplan]

What is the status of this release? I have an app that uses Bullmq and want to migrate to fly.io and use upstash redis on fly. It would be nice not to have to change my queue implementation

[mdogan]

We are currently deploying the fix to all regions gradually, not sure whether it's deployed to fly.io regions at the moment.

But note that, this fix is only for older Bull library (specifically to support dynamic keys in Lua scripts), not for the new BullMQ. BullMQ uses Redis Streams to publish events but Upstash doesn't support streams api yet.

[antoinedc]

We are currently deploying the fix to all regions gradually, not sure whether it's deployed to fly.io regions at the moment.

But note that, this fix is only for older Bull library (specifically to support dynamic keys in Lua scripts), not for the new BullMQ. BullMQ uses Redis Streams to publish events but Upstash doesn't support streams api yet.

Any plans to support it?

[manast]

@antoinedc events in BullMQ are not a critical feature, meaning the queues will work without the events. Of course, the events may be important for you for debugging purposes, etc. If streams do not land in upstash, it is possible that we can have an events system based in pubsub again, as we had in Bull.

[antoinedc]

@antoinedc events in BullMQ are not a critical feature, meaning the queues will work without the events. Of course, the events may be important for you for debugging purposes, etc. If streams do not land in upstash, it is possible that we can have an events system based in pubsub again, as we had in Bull.

Hi Manast, thanks for the reply. I'm confused now 😅, does that mean that BullMQ works on Upstash then?

[manast]

I don't know... I am just saying that events are not critical so even if streams are not implemented in Upstash, it could work.

[mdogan]

We are currently deploying the fix to all regions gradually, not sure whether it's deployed to fly.io regions at the moment.
But note that, this fix is only for older Bull library (specifically to support dynamic keys in Lua scripts), not for the new BullMQ. BullMQ uses Redis Streams to publish events but Upstash doesn't support streams api yet.

Any plans to support it?

Streams are in our roadmap but not in the short term unfortunately.

[mdogan]

@antoinedc events in BullMQ are not a critical feature, meaning the queues will work without the events. Of course, the events may be important for you for debugging purposes, etc. If streams do not land in upstash, it is possible that we can have an events system based in pubsub again, as we had in Bull.

Hi Manast, thanks for the reply. I'm confused now 😅, does that mean that BullMQ works on Upstash then?

No, it doesn't work at the moment. Because any call like redis.call("XADD", ...) or redis.call("XTRIM", ...) raises an error and script execution is terminated at that point. As an alternative, if they are called via redis.pcall(...) instead, then script will continue to run and return error reply for these unsupported commands.

I don't know the all use-cases for BullMQ, but it might be easier to optionally turn off events feature than implementing them based on pubsub.

[chronark]

Streams are unlikely to be available on upstash in the next few months, so let's discuss options:

1. replace stream calls with pcall to partially ignore errors and partially handle them gracefully.
2. allow the user to disable streams
   this would require adding a new argument to lots of scripts and pass that configuration around.
   Sounds like a lot of spaghetti to me.
3. Event system build on pubsub as mentioned here

@manast I'd like to work on this and submit a PR to bullmq, but I would appreciate your opinion on what's the best approach.

[rizexor-endeva]

yes high command usage. use redis cloud instead

[kimitrii]

I'm almost contract a plan in upstash but my code use BullMq and when I saw one simple queue with two items generate more then 200 commands and keep counting after the execution, I literally gave up. It's make no sense? What the logic of it?

[manast]

Currently, the biz model of Upstash does not fit well with BullMQ, even if it's supported and works reliably. We are trying to minimize the number of commands issued but still, there are going to be a lot of commands per job unfortunately as there are a lot of things to book keep.

[navaneeth-dev]

I'm almost contract a plan in upstash but my code use BullMq and when I saw one simple queue with two items generate more then 200 commands and keep counting after the execution, I literally gave up. It's make no sense? What the logic of it?

use fast api with python. it has inbuilt background tasks

[mdogan]

Currently we (Upstash) are gathering statistics about empty commands (write commands which are not mutating the state and read commands returning empty response or null) generated by libraries like BullMQ, Sidekiq etc. We are discussing & considering not to account these commands, but it's not decided yet.

[barbinbrad]

Currently we (Upstash) are gathering statistics about empty commands (write commands which are not mutating the state and read commands returning empty response or null) generated by libraries like BullMQ, Sidekiq etc. We are discussing & considering not to account these commands, but it's not decided yet.

Thanks for the update Mehmet! I'm sure many are glad to hear that.

[MartinL83]

Currently we (Upstash) are gathering statistics about empty commands (write commands which are not mutating the state and read commands returning empty response or null) generated by libraries like BullMQ, Sidekiq etc. We are discussing & considering not to account these commands, but it's not decided yet.

Is there any update on this @mdogan ? I would like to use Upstash with BullMQ. I tested out Bull yesterday with 2 queues and 100 something jobs. That ate up 10k commands in one go. So its not realistic to use BullMQ currently with Upstash :/

[TheRealFlyingCoder]

Just jumping in here, as the Fly.io documentation now explicitly says:

Note that empty responses are not counted towards your bill. This prevents billing for standard polling behavior of tools like Sidekiq or BullMQ.

Although i've just deployed an app with BullMQ, with no jobs and 1 queue but it chewed up all 10K daily requests? 🤷

[manast]

@TheRealFlyingCoder BullMQ does not do a lot of polling, and in fact you can reduce it as much as you want with the drainDelay option (https://api.docs.bullmq.io/interfaces/v4.WorkerOptions.html#drainDelay). The problem is the amount of commands executed every time a job is added and processed, could be something between 10 and 20 commands per job.

[manast]

Although i've just deployed an app with BullMQ, with no jobs and 1 queue but it chewed up all 10K daily requests? 🤷

Do you have a log with the commands you were charged for that empty queue?

[mdogan]

Just jumping in here, as the Fly.io documentation now explicitly says:

Note that empty responses are not counted towards your bill. This prevents billing for standard polling behavior of tools like Sidekiq or BullMQ.

Although i've just deployed an app with BullMQ, with no jobs and 1 queue but it chewed up all 10K daily requests? 🤷

Normally empty responses are not billed but due to a bug they are still counted in the free-tier quota at the moment. The fix is being gradually deployed to all AWS and fly.io regions.

Still, not all commands are considered empty while BullMQ is idle. Especially EVAL/EVALSHA commands are counted (but not the empty commands embedded inside Lua script).

I did a quick test with a single worker and single queue in 5mins period. Here are the metrics:

 "DailyProcessedCommands": 445,
 "DailyWriteCommands": 184,
 "DailyReadCommands": 261,
 "DailyEmptyCommands": 335,

445 commands (184 writes, 261 reads) are sent by the idle BullMQ worker, and 335 of them are considered empty and not counted as billable. So still, 25% of the idle requests are billed.

These billed commands are mostly EVALSHA and some occasional SET and HGET calls.

[TheRealFlyingCoder]

Normally empty responses are not billed but due to a bug they are still counted in the free-tier quota at the moment. The fix is being gradually deployed to all AWS and fly.io regions.

Just to confirm @mdogan all 445 would have been counted as billed requests currently in Free Tier on fly due to the bug?

But once a fix is implemented 25% of idle requests will still be counted towards the daily limit?

[CanRau]

Very exciting this is being figured out, probably returning to upstash soonish then 🥰

Thanks a lot for all the work ❤️

[TheRealFlyingCoder]

Just checking in on the status of this... was there any work that we needed to do for Fly.io instances?

I have a development server online, with zero activity on it, and upstash is currently averaging 35K requests/day

So far I have 300K billable requests, 200K empty requests showing up on the usage for the month.

[manast]

@TheRealFlyingCoder which request is the one producing the empty 200k requests, BRPOPLPUSH by any chance?

[buggyhunter]

@TheRealFlyingCoder we would like to check deeper if you provide your database name?

[TheRealFlyingCoder]

@TheRealFlyingCoder which request is the one producing the empty 200k requests, BRPOPLPUSH by any chance?

So unless i'm crazy I can't find any way to check logs with Fly.io Upstash?

@buggyhunter it's 'theprintbar-dev-redis'

[mattapperson]

Just checking in with this. Has the issue of being charged for empty requests / using BullMQ very large queues been solved?

[TheRealFlyingCoder]

Just checking in with this. Has the issue of being charged for empty requests / using BullMQ very large queues been solved?

Not that i'm aware of... i had to switch to a paid subscription just to keep it alive.

My BullMQ with no activity (just polling) hits exactly 166K requests per day every day, of that ~50% are put down as billable

[manast]

I think you can change the "draingDelay" setting to something much larger, like 60 seconds or so, that would reduce the requests to "just" 13k or something. I hope I can work something up with the upstash team to make this much better.

@TheRealFlyingCoder BullMQ does not do a lot of polling, and in fact you can reduce it as much as you want with the drainDelay option (https://api.docs.bullmq.io/interfaces/v4.WorkerOptions.html#drainDelay). The problem is the amount of commands executed every time a job is added and processed, could be something between 10 and 20 commands per job.

Did you try this?

[TheRealFlyingCoder]

Did you try this?

Will push an update now and let you know if I notice a drop... I didn't update it originally because the wording seemed to suggest it wouldn't be the problem haha

At a default of 5 seconds it still shouldn't be hitting 160K/day unless i'm misunderstanding how it works

[lildesert]

I'm just sharing my experience here but while we're waiting for this to be fixed we deployed a redis on a fly.io app of our own following this repo https://github.com/fly-apps/redis/.
It's been working well for our usage so far (which is pretty basic) and it's much more cost-efficient.

[jorisvdz]

Upstash launched a new project (https://github.com/upstash/queue), maybe this will be of interest to some of you?

[TheRealFlyingCoder]

Did you try this?

@manast Just coming back to this, Drain delay saw no improvement... and as a follow up I've continued to build out my solution which involves more queues and it's increasing exponentially.

I am now up to 2-3M requests per day with zero traffic hitting the server 😬😬😬

All of the graphs are flat, i'll share the stats and hopefully you'll see if anything stands out:

Daily Commands: 2m - 3m
Daily Bandwidth: 150MB - 200MB
Throughput: 20 Reads & 16 Writes per second
Data Size: 24.38KB
Connections: 138
Keyspace: 85
Hits/s: 37
Misses/s: 97

[manast]

@TheRealFlyingCoder if you want you can contact me via email and provide more details of your solution so that I can understand better where all these calls are going.

[manast]

I just made some tests with @TheRealFlyingCoder example code and with 1 worker it should consume around 700k commands per month, which at current costs is less than 1.5$/mo. There are a couple of things to note here however, I am assuming only "write" commands are charged, and I think that should be the case. Secondly, it would be even better if Upstash only charged for commands that actually changed the database. So for instance, if I delete an un-existing key, no charge for that command... If that was achieved, then for an empty queue there is by default only 1 write command every 30 seconds, or basically zero costs.

[TheRealFlyingCoder]

I'm also keen to know whether my solution has a flaw in it's implementation or whether that polling quantity is to be expected.

Because as I stated before it is ramping up a lot, I am up to 17 queues, running on each of my 3 dev servers.

Using @manast's 700K requests and 51 active queues, I would expect 35M writes/month now (~1M/day which is what i'm seeing in the usage) and a base operating cost of ~$75/month just from the write command polling.

The read commands are much higher and also seem to be charged though but I don't have the numbers on those (Fly.io upstash you can't see any real info)

[vincentdoerig]

For those using Upstash with fly.io, they just announced a $10/month fixed plan with no cap on the number of commands executed: https://community.fly.io/t/upstash-for-redis-new-10-mo-single-region-fixed-price-plan/19555.

[enesakar]

We have updated our billing to make BullMQ/Bull/Sidekiq usages more affordable.

Docs: https://upstash.com/docs/redis/integrations/bullmq
Announcement: https://x.com/upstash/status/1825622313636868230