Intended use-case for flows

[rosslavery]

Hi there.

Currently using Bull and looking to migrate to BullMQ.

We currently have an "init" job that dispatches thousands of smaller "process" jobs, which collect a small snippet of data and then save it to a database. We are running into DB performance issues due to the large number of many small inserts happening inside the "process" jobs.

I'm interested in moving to BullMQ to take advantage of the flow system, where our "init" job would be a flow producer that has N child jobs (~10,000) that run in parallel. After the child jobs finish, the "init" job would gather the results of the child jobs, and save to the DB in batches, to avoid hammering the DB with many small inserts.

Is this a valid use case for flows? Is there a better approach others have taken?

The docs are understandably simplified and small in scale (car, engine, wheels), but wasn't sure if there are architectural limitations on a "reasonable" number of child jobs that can be inside a flow, and whether the "getter" helpers would choke gathering results from that many children.

[manast]

There should not be any performance issue regarding the amount of child jobs, since they will be executed in a queue as any other job, so with a proper amount of workers and concurrency they should be processed efficiently. It will be important to make sure failed jobs are handled properly as a single failed job will prevent the parent job to execute.
Maybe @roggervalf has more input/feedback to give as he has more experience running flows in production than I do.

[manast]

Btw, have you seen this blog post? https://blog.taskforce.sh/splitting-heavy-jobs-using-bullmq-flows/