Need help for best practice on a multi-server set up.

[hagginoaks]

Server 1: hosts my node app and has a connection to my DB with all my models. No connection to bullmq or redis.
Server 2: receives POST requests with job details through a REST endpoint from Server 1 using node-fetch whenever a job needs to be added to my worker. Server 2 has connection to redis and bullmq where it processes the jobs but no access to my models or DB.

My question: After jobs are processed on Server 2, should I send a POST request BACK to Server 1 since it has access to the DB and all models? Or should I duplicate my models and DB connection on Server 2 so Server 2 doesn't need to "report back" to Server 1 to complete the job?

Alternatively: Should I just throw bullmq and redis as more dependencies on Server 1 and handle everything on it and remove Server 2 from the equation?

Thank you.

[bilalshaikh42]

Hi,
We are using a similar setup at BioSimulations.

We have an API (server 1 in your case) that has access to the database and models. Then we have a service (server 2 in your case) that does a lot of the processing. We have set it up as follows:

The API contains a "dispatch" queue where it adds jobs. These jobs are then received by the service using the queue and the service does the processing. Then, the service sends a POST to the API with the results of the work. This is working very well for us.

I like this approach for a few reasons.

1. The CPU intensive processing is kept separate from the API. This allows us to scale up our service while keeping the API constant.

2. The API is the single source of truth for the state of the DB and models. Whichever approach you use, I would highly advise against having multiple services have access to modify the database. That is inviting issues with concurrency and causes a lot of trouble following the changes in state. You can read more about this here

3. The API can handle a large number of requests without getting overloaded. All it needs to do is upload the database and then add a job to the queue. If there are a large number of jobs, the queue will increase in size, but the load on the API won't.

[hagginoaks]

Hi @bilalshaikh42 thank you very much for the reply here and helping me with my question. Your website and repo is way more professional than mine so please bear with me as I try to understand better.

For clarification:

1. Your API service has to do some work.
2. It is always connected to your redis instance and when it does the work, it pushes work data to a preliminary dispatch queue.
3. Service (server 2) is also connected to and is listening to that dispatch queue
4. Service does all the additional processing and then pushes the final result to your API server

If this is correct, then in your API server, do you have a special route that processes all incoming requests from the service? I'm thinking of how to do this securely.

Here are some example flows that I'm building:

User password reset:

1. User requests password from API server
2. API server pushes request with job data to dispatch queue
3. Service (server 2) picks it up and gets to work (e.g. pushing it to an email queue)
4. Email queue gets processed and POSTs the data back to my API server at the endpoint api.mydomain.com/taskworker/email
5. My /taskworker/email endpoint processes the incoming POST request which calls an email controller that imports my sendgrid library and takes the incoming job data (from Step 2) and sends the email to the user.

I think this flow is easy for me to understand, but I get confused when I need to do a cron job. Example below:

Scraping job:

1. Every 12 hours, my dispatch queue will trigger a scraping job.
2. It will POST to Server 2 with the type of job that it needs to process.
3. Server 2 then will POST to an endpoint on my API server api.mydomain.com/taskworker/scrape with the type of job that needs to be handled
4. It will scrape the data and then push that back to the dispatch queue, thus repeating the life cycle

Does this logic sound correct for the scraping job? Thank you so much i'm really starting to understand the power of all of this setup.

[bilalshaikh42]

If this is correct, then in your API server, do you have a special route that processes all incoming requests from the service? I'm thinking of how to do this securely.

Yup, you got it. Our API is a standard REST API, with different routes representing different resources. So /models for models, /logs for log objects etc. The service calls the appropriate route based on the task that it is doing. In terms of doing this securely, that is really a different question from the architecture. It is possible to secure endpoints in various ways, the preferred standard being the use of JWTs. So our service uses a client_id/client_secret to get an auth token that it uses to authenticate to the API

User password reset:
1. User requests password from API server
2. API server pushes request with job data to dispatch queue
3. Service (server 2) picks it up and gets to work (e.g. pushing it to an email queue)
4. Email queue gets processed and POSTs the data back to my API server at the endpoint api.mydomain.com/taskworker/email
5. My /taskworker/email endpoint processes the incoming POST request which calls an email controller that imports my sendgrid library and takes the incoming job data (from Step 2) and sends the email to the user.

While this would work, I don't know enough about your use case to comment on this. If all you are doing is sending an email, why not have the service just send the email directly? Simmillarly, rather than having a separate dispatch and email queue, you could just add a job to the email queue directly.

Does this logic sound correct for the scraping job? Thank you so much i'm really starting to understand the power of all of this setup.

So BullMQ has the concept of repeatable jobs. Instead of managing the repeat manually, does this sound more like what you are trying to achieve?

Regarding step 2, "POST to server 2" , you can just have server 2 directly listen to the queue for those jobs. Unless you really need to have only one queue attached to the API, you can just put the appropriate job (mail/scrape, etc) into the appropriate queue and have the service pick them up directly. In general, you can have many different types of jobs and queues without needing a single "dispatch" queue to go through. In our case, we used a single queue to communicate between the API and service, but that is only because that is what made sense for our particular use-case. The API is only really aware of one type of job that needs to be done.