Benchmark speed of scheduling (Agent)

[frosk1]

Hi all,

I am really liking the project. It would be great also for better industry adoption to present some benchmarks for the scheduling speed of the prefect agents vs other common scheduling frameworks.

Compare to kubernetes cron jobs, local cron agents, airflow etc.

[zanieb]

Could you explain what metric you're thinking of measuring to get "scheduling speed"? Flow runs are scheduled by a dedicated service and creating a scheduled run is unlikely to ever be a bottleneck.

[frosk1]

hey @madkinsz it is more about the time the scheduler needs to start a job. *sry if I am not using the correct wording here.

So assume the following setup:

running WebUI, running prefect server (apollo service) , running Agent process

The agent process is responsible for starting/running tasks/jobs/flows .

Now people would like to know how fast is the agent process in starting the tasks:

• how fast tracks the agent that now is the time to start the task, by communication with the server backend
• how fast are docker images loaded and run
• how fast is a kubernetes agent starting the jobs
• ...

The speed performance means:

if I schedule a task/job/flow at timestamp 10:01:00.00
and the tasks actually starts at timestamp 10:01:00.50 or at 10:01:01.00

we would know how long it takes an agent to start a task.

Was this clear enough?^^

If prefect can not run flows/jobs/tasks within a second, this might influence high frequency critical use cases.

[zanieb]

We're not in control of the time it takes for the job to start running on your Kubernetes cluster or for a Docker container to start. These are dependent on lots of other factors. A simple example is the size of your runtime image and whether or not it has been pulled already can have drastic impacts on the amount of time it takes for the container to start running. At this point, you wouldn't be benchmarking Prefect, you'd be measuring the specific Kubernetes or Docker setup which I don't think provides much value. We can measure the scheduled time -> submitted time at which point the job has been submitted to external infrastructure like Kubernetes, but Prefect adds very little overhead there.

[frosk1]

Do you understand that I am aiming for high frequency and low latency use cases where those triggers to start a scheduled job should definetly being under 1 second

[zanieb]

The agent polls every 10 seconds (at most, after not finding any runs) and when it polls it looks for runs scheduled in the next 10 seconds to avoid late starts. When it finds runs, it submits them concurrently in threads. So, worst case scenario is a run gets created during the polling interval so it is missed until ~10 seconds have passed. Realistically, your runs will be pre-scheduled and will be submitted immediately. Any additional overhead in this case is from contact with the service your flow run is being submitted to, which will be dependent on your environment. Prefect should not be a limiting factor in this instance.

Yeah I understand. Prefect is not intended for sub-second latency as most of the tooling we integrate with is slower.

[frosk1]

Could you explain what the happens in the thread? I guess the thread checks against time until the minute is reached after 10 secs and then the thread triggers the flow, is this correct?

[zanieb]

Yep the thread waits until the scheduled start time then submits the job to the execution infrastructure e.g. creates a container which then starts execution of the flow.

In v2, we're investigating pre-creating the execution infrastructure before the scheduled start time with a configurarable "warm-up" period so the flow run can begin exactly on time.

[frosk1]

@madkinsz thx for clarification.