BullMQ on multiple computers

[eltoroit]

I have a long process that is made of 5 steps. The steps need to be executed on the same machine where the process started. Independent processes could be executed on different computers for scalability.

Let me give you an example.

I have this long process that is made of five steps: A-B-C-D-E
I am running my application on two computers: 1 and 2.

When the process executes on computer 1, I will be doing steps A1-B1-C1-D1-E1.
When the process executes on computer 2, I will be doing steps A2-B2-C2-D2-E2.

The steps could take a long amount of time to complete because I need to wait for other servers to do their job, so when BullMQ wakes a particular job, I look at which step needs to be done and I execute that step once the step is completed, then I put the job to sleep for 30 seconds.

I must prevent this situation: A1-B1-C2-D2-E1.

How can I do that?

[manast]

Since jobs are workers agnostic you cannot bind a set of jobs to a particular worker in BullMQ. I am not sure if you know a about flows? maybe they are useful to you somehow: https://docs.bullmq.io/guide/flows

[eltoroit]

If I use flows, will I be able to lock the children to run on the same machine as the parent?

[eltoroit]

Flows look better than the approach I took for breaking the process down.

This is what I am currently doing inside a single machine, and it's working fine. I am not sure if this is a hack and prone to fail in the future (I can't find this documented)

The function that is used on the instantiation of the Worker, will receive the job to be executed and I return a percentage of completion depending on the step that it's executing. If we keep with the example I gave above (A-E) then each step returns an additional 20%. This makes the job go back to the queue until it completes when the progress reaches 100%.

[eltoroit]

I have an idea. What if I have several queues, one for each machine? When the job needs to be started, I will add it to the queue (machine) with the least number of active jobs. The workers on a single machine will only be watching one queue, the one for that machine.

Will that work?

[dschnare]

What about using suffixed job names to indicate the machine type? So the first enqueue operation could be for job "a", then the worker that processes this job would then enqueue "b/1", that would only be processed by the same machine.

So machine "1" can process "a", "b/1", "c/1", "d/1" and "e/1".
Machine "2" can process "a", "b/2", "c/2", "d/2" and "e/2".

That way your application code can enqueue "a" without caring about the rest of the workflow. However, this would require you enqueuing jobs in sequence from one worker just before it's completed, and not part of Flow.

[dschnare]

Hmm. I guess that will be challenging because you'd still need to listen to jobs on the same queue and do a switch based on job-name. Unless it's easy to re-schedule a job if a worker starts processing a job that it can't handle you'd have to use multiple queues.