How to manage the "Manually processing jobs" approach with multiple workers?

[geosigno]

I build orchestrator/transcoder applications. For the big picture:

• the orchestrator will receive several encoding tasks from the front-end application
• it will distribute the jobs to the worker pool (GPU instance managed on a Kubernete cluster)
• it is also responsible for running the create GPU instance command / and receiving events once they are ready to serve the job
• orchestrator and transcoder will use a common Redis to share queue information

So I have to use the "Manually processing jobs" approach because when the orchestrator has received a task, it does not mean that a worker is ready, because the minimum pool size is 0 (for cost reasons) and there is also a max poll size. limit. And each worker will only have one job concurrency at a time, as it will use a lot of GPU resources through ffmpeg.

I'm looking for documentation/examples on this, but can't find anything.

• Does the approach of having a single queue and multiple workers (via kubernetes) make sense for my use case?
• How do I know if a worker is already working? At the moment the only way I have is to do a queue.getActiveCount(), but that gives me all active jobs in the queue (which is unique) and not per worker. Do I need custom logic built on my side and store information in Redis for example.

[manast]

Having many workers consuming jobs is the standard use case of BullMQ. Not sure I understand why you need to use manually fetched jobs since you can configure which concurrency level you want to use in every worker (being 1 the default).
Here there is a blog post about transcoding that maybe is useful to you: https://blog.taskforce.sh/splitting-heavy-jobs-using-bullmq-flows/