DatabricksRunMultipleNow class

[bjeffrey92]

In the prefect.tasks.databricks module there are 3 classes:

• DatabricksRunNow which runs an existing databricks job that is identified by it's databricks job id
• DatabricksSubmitRun which creates and triggers a one time run of a new job (forgive me if this is an inaccurate description, I haven't actually used this class before)
• DatabricksSubmitMultitaskRun which allows the user to create and trigger one time runs of multiple new jobs

I would like to be able to run multiple existing databricks jobs in a single prefect task. Is there a reason why there isn't a DatabricksRunMultipleNow that would take as input a list of databricks job ids (and optionally some configuration for these jobs), and start all of these jobs in parallel returning success or fail based on whether or not they all complete?

If not, would the community be open to creating something along these lines in prefect?

Alternatively, please do correct me if you believe that what I am describing can be achieved easily with the existing module

Thanks