Multiple worker instances per node process?

[officert]

My previous setup with BullMQ was to instantiate a new Worker instance per type of job I was trying to run. For example job 1: send email, job 2: process image, etc...

So a single Node process would end up instantiating a lot of Worker instances (around 30). Eventually I started to see some strange Redis errors like

ReplyError: Too many requests. Please try again later.
Mar 30 05:21:41 PM      at parseError (/opt/render/project/src/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/li

In reading the docs more closely it seems this is the wrong way to use Bull MQ and instead we should be creating a single Worker instance per Node process and have the 1 worker's job handler do a big switch statement.

Am I understanding things correctly? Does it ever make sense to have more than one instance of Worker per Node process?

[patrickbolle]

I am having this same error message popup, but it seemingly is showing despite my code not changing for 6+ months. Did you figure this out any further @officert ?

[officert]

@patrickbolle yes, in my case I was instantiating quite a few (20+) worker instances. Essentially I had a worker instance per channel. So I refactored my code so I'm only creating a single worker instance per node process (1 instance in my main web process, and 1 instance in a worker process).

This also seemed to pop up all of a sudden for me, as it was working for awhile. My guess is I continued to add more "channels", which mean additional worker instances and each worker is probably connecting multiple times to Redis and I hit some kind of cap.

Another issue I had been running into prior to my refactor is I would get jobs that would just get stuck occasionally. They would run and I could watch the progress on them, but they would hit about 99% complete and just never move to the completed queue.

Now that I'm no longer creating multiple worker instances I'm not seeing this issue at all anymore. This leads me to think that you're really only supposed to create a single worker instance per node process, but the docs aren't very clear about this so I posted this question hoping for some clarity.

[manast]

This is a very strange error that I have never seen before and I wonder if it is something ioredis produces in some edge cases 🤔

[officert]

@manast Could you shed some light on my original question.....should you ever have more than 1 Worker instance per Node process? I can try and put together a small test to reproduce the error. It seemed to happen after having 20-30 Worker instances all in a single Node process.

[manast]

It should not be a problem to have as many worker instances as you want per Node process, but of course all depends on what are the workers doing, as they will consume memory and CPU.