Fan Out Jobs

[fotoflo]

Hi everyone, I was checking out the Quirrel noticed that there is mention of fanout jobs on the home page, but I couldn't find any documentation about them.

I was able to get it a cron job working on my local host, but of course that doesn't create fan out jobs.

Does this automatically fan out on serverless functions or do i use the exclusive peremeter or something?

Any insights or resources would be greatly appreciated. Thank you!

[Skn0tt]

Hi @fotoflo! Sorry for the late reply, I was on PTO :D

To implement a fan-out job, you can create a Queue and then call .enqueue or .enqueueMany on it, to create jobs that will be processed in other serverless functions.

[Skn0tt]

Assuming this is also the right thing to do for fan out jobs?

Depends on what you're trying to achieve. Using exclusive will mean that there's only one job run at the same time, now matter how many jobs there are. For something long-running like lighthouse reports, that can become a bottleneck. Could you elaborate as to why you're intending for them to run exclusively?

[Skn0tt]

Will be deploying soon so can't wait to see it in action.

Just to make sure you don't hit a wall, because people have been bitten by this before: There's no hosted offering anymore, and you'll need to stand up your own Quirrel instance.

[fotoflo]

Thanks yeah I see that. I was thinking to setup a upstash redis instance and then it looks like I just drop the api keys/database uri in vercel, but not sure about how to use the quirrel ci command... will investigate

[fotoflo]

Assuming this is also the right thing to do for fan out jobs?

Depends on what you're trying to achieve. Using exclusive will mean that there's only one job run at the same time, now matter how many jobs there are. For something long-running like lighthouse reports, that can become a bottleneck. Could you elaborate as to why you're intending for them to run exclusively?

This is how I have to do it on local... will the jobs fan out if I use exclusive in vercel?

[Skn0tt]

will the jobs fan out if I use exclusive in vercel?

No, they won't. The exclusive option is for operations that strictly require sequential processing, like some usecases in payment + order fulfilment.

I was thinking to setup a upstash redis instance

Let me know if that works well - my understanding is that Upstash is a custom redis implementation, and i've never tested it with Quirrel. You'll still have to host the Quirrel docker image somewhere, though.