Large number of delayed jobs with the same delayedTimestamp

[willemchua]

Hi everyone, I'm new to BullMQ, and I'm hoping to use the delayed jobs feature in one of my projects.

So, as I understand it, when we add jobs to the "delayed" redis queue, it will add the jobId as the entry, with the score being delayedTimestamp * 4096 + (jobCounter & 4095) (reference). So, this means that after 4095 jobs with the same delayedTimestamp, the order of the jobs wouldn't be retained.

For example: delayedTimestamp = 10000 (for simplification)

Job 1 - 10001
Job 2 - 10002
...
Job 4095 - 14095

After job 4096, it becomes

Job 4096 - 10000
Job 1 - 10001
Job 4097 - 10001
Job 2 - 10002
...
Job 4095 - 14095

Is there any way to store more than 4096 delayed jobs with the same delayedTimestamp while still retaining the order of when the jobs are created? Initially I tried to set an epoch and the first part of a random uuid (ex: 1647545149603-cbd80f7b) as the jobId so that the job order would be retained lexicographically (cmiiw, but I understood that to be how zrange in redis work).

Please correct me if I'm wrong if this is not the actual behavior, and if it is, is there any way to retain the order of large number of delayed jobs with the same timestamp?

[manast]

So you are planning to add more than 4096 jobs per millisecond or in order words 4M jobs per second? If so, and the order is critical then yes this will be a problem. But I am quite sure you won't be able to reach this limit.

[willemchua]

Hi @manast, thanks for the prompt reply.

Actually, yes, that is part of my use case. So what I'm trying to do is to queue jobs to be processed at a later date, with the date being user-specified. For the most part, the dates are not overlapping, but there are regular cases where there's a burst of request pointing to the same date (and therefore, same millis). Although I'm can't say for sure what the exact number of requests will be, I'm expecting it to be more than 5k for that same millis.

[manast]

The problem is that you will not be able to add 5M jobs per second. When you add a delayed job Bull will sum the current timestamp + delay. The only way you will be able to produce more than 4k jobs in the same millisecond would be if you add the jobs in 1 batch. In that case, just divide the batches in less than 4k per batch and you will be fine. Still, with so many delayed jobs it will be extremely difficult to give guarantees about the processing order of the jobs, not to mention you will need a lot of parallel workers which of course will run some jobs out of order.

[willemchua]

The problem is that you will not be able to add 5M jobs per second.

The way I use BullMQ delayed param is to subtract the current epoch with the expected epoch new Date().getTime() - expectedDate.getTime(). So, this is how I got > 4096 jobs in a targeted millisecond.

Still, with so many delayed jobs it will be extremely difficult to give guarantees about the processing order of the jobs

Yep, I've acknowledged this. However, I'm trying to see if I can still maintain the relative order of the jobs, even though it might be processed slower and there are some jobs that might be processed out of order. I'm trying to avoid having the (hypothetical) 8000th job be processed first in the worker.

But, I also acknowledge that this might be very specific to the problem I'm trying to solve and not something that BullMQ has in the future pipeline. I'll try to arrange some process on top of the BullMQ to achieve something close to this.

I highly appreciate the answers! Thanks @manast 🙇