Idea for job triggering?

[HarikalarKutusu]

FlowJobs have a top-down approach, you want a final result and you define children to reach that. I have a scenario that needs the reverse logic:

• A worker checks a remote resource and compares that with a locally cached one. If no change, all is well.
• If the resource changed, the worker returns { status: "updated"}. In this case a chain of other events (a tree) should be triggered.

For example (I'm dealing with datasets):

• The original resource is a json with language codes.
• If the resource changed, the following chain should be triggered:

JOB: For each language in languages:
  JOB: Download each language dataset (no parallel - they might be large)
   => JOB:  Expand downloaded tar.gz
        => JOB:  Process it (statistical analysis etc)
   => JOB: Delete files
Combine results and save.

I had that implemented using bree using a list of job definitions and a JOB_CHAIN construct, but started to move to redis/bullmq for inter-microservice communication and caching. I was hoping o use Flows after an initial read, but it does not seem to fit.

As I'm pretty new to bullmq (2 days), any idea to approach the problem will be very much appreciated.

Edit: I missed that the repo has a Discussion section, please feel free to move this there. I also found the following related discussion:
Flow Producer Design for cases where children jobs are dynamically defined by another worker

[manast]

I think BullMQ can solve that use case already without any new features. Have you read this blog post?
https://blog.taskforce.sh/splitting-heavy-jobs-using-bullmq-flows/

[HarikalarKutusu]

Yes @manast, this one and others... I concluded that I need a more general-purpose "orchestrator" at a higher level. I had to implement a rudimentary one as I explained here:
#1858 (comment)

Current children-children-children... type chaining suggestion, without a conditional flow logic can only handle a small part of possible scenarios. Adding a simple orchestrator feature to BullMQ, would help to solve many cases.