How to specify that a job should only run after a certain point in time

[SteVwonder]

Ultimate goal: emulate Slurm's -b, --begin=time flag

This requires:

• a new arg/flag for either flux mini or flux run (once that is out)
• a well defined key within jobspec
• updates to fluxion-qmanager and/or fluxion-resource to read that key within jobspec and enforce it

Beyond just delaying user submissions to a time more conveinent to the user, this could also be useful for settings up large reservations, DATs, system downtimes, etc.

[grondo]

Just to start the discussion with a dumb idea: could the minimum start time be considered a job dependency, and thus be enforced by the job dependency service? This would potentially allow --begin-time to be supported by any loaded scheduler.

[SteVwonder]

Yeah. I really like that idea! I think that would work brilliantly for the user-level case. For things like DATs, I think we'd want to go a different route to ensure the time is respected exactly, but that's not really the point of this discussion anyways.

If we go that route, then maybe we add time as a valid type and URI scheme to RFC 26? It would also minimize the changes we need to make to flux mini since we can piggyback off of the dependency specification (once that is in place).

[grondo]

I was just re-reading the Slurm documentation of --begin and there is a subtlety I missed at first:

-b, --begin=<time>
    Defer initiation of this job until the specified time. 
    ....
    The job will be eligible to start on the next poll following the specified time. The exact poll interval depends on the Slurm scheduler (e.g., 60 seconds with the default sched/builtin).

From this, it sounds like srun --begin sets a time at which the job will run (+60s), not a time before which a job should not run. If we indeed need to support the former case, then using a time dependency will not work, since there is no guarantee when the job will actually be started after the specified time. Supporting an actual begin time will require support from a scheduler.

To me, it seems like allowing users to specify an exact begin-time isn't wise, so perhaps I'm not understanding the docs, though.

[dongahn]

I am surprised. This means slurm uses --begin as its advanced reservation. But the slurm man page has no indication of such. They have a whole new section on AR (https://slurm.schedmd.com/reservations.html) which doesn't mention this option at all.

The exact poll interval depends on the Slurm scheduler (e.g., 60 seconds with the default sched/builtin)

Pretty badly written, but one way to interpret this is "this scheduler will consider job's scheduling eligibility within up to 60s (in the case of default conf) after <time>? Similar to your approach?

[grondo]

My interpretation is the same as yours, but I am unsure of what the real requirement is here.

My intuition is that we should support a "minimum begin time" as proposed here, but I wonder if also any existing Slurm users are using --begin to more precisely schedule job run time, though that seems like a feature that could easily be abused.

Probably @ryanday36 will know the correct answer here.

[dongahn]

using --begin to more precisely schedule job run time

If I were to do this in Fluxion, I would get the jobspec right away and create an "allocation" into the future in the next scheduler loop invocation. If I can't schedule it, I would reject it immediately. Having an allocation into the future will significantly impede "backfilling" operations and reduce the overall system throughput so I would factor this into user's priority...

This would be better to be exposed as --advanced-reservation=<time> then --beginat a minimum. And We also probably want to limit the user classes who can use such an option...

[ryanday36]

I think that by 'eligible to begin' they mean something like 'considered by the scheduler', so more like the job is in a 'held' state until the time given by --begin. It's not a guarantee that the job will begin at a certain time. It's a guarantee that a job will not start before that time. The main use case is that there is a scheduled down time, DAT, etc. and the user doesn't want to take a chance on the job getting killed early, so sets --begin to some time after the down time or DAT is scheduled to start.

[grondo]

This would be better to be exposed as --advanced-reservation=<time> then --beginat a minimum. And We also probably want to limit the user classes who can use such an option...

I completely agree. Let's find out if this is even a requirement for initial system instance

…

[ryanday36]

This would be better to be exposed as --advanced-reservation= then --beginat a minimum. And We also probably want to limit the user classes who can use such an option...
I completely agree. Let's find out if this is even a requirement for initial system instance
…

I have it in the production dev (rzgenie, rzalastor) column, but it could probably be bumped down to production compute (quartz, rztopaz).

[grondo]

Thanks Ryan! Sorry, it seems like I was confused by the docs.

…

On Tue, May 11, 2021, 12:10 PM Ryan Day ***@***.***> wrote: I think that by 'eligible to begin' they mean something like 'considered by the scheduler', so more like the job is in a 'held' state until the time given by --begin. It's not a guarantee that the job will begin at a certain time. It's a guarantee that a job will not start before that time. The main use case is that there is a scheduled down time, DAT, etc. and the user doesn't want to take a chance on the job getting killed early, so sets --begin to some time after the down time or DAT is scheduled to start. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3243 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFVEUUOLIHPMY3X4LP3J43TNF6L7ANCNFSM4SGZFVSQ> .

[grondo]

I have prototype support for this (as a test for multiple jobtap module implementation):

$ src/cmd/flux start sh -c 'flux jobtap load builtin.depend.begin-time; flux mini run -vvv --setattr=system.begin-time=$(($(date +%s)+10)) hostname'

jobid: ƒS9ppJK
0.000s: job.submit {"userid":1000,"urgency":16,"flags":0}
0.013s: job.dependency-add {"description":"begin=2021-05-12T20:37:09 GMT"}
8.782s: job.dependency-remove {"description":"begin=2021-05-12T20:37:09 GMT"}
8.782s: job.depend
8.782s: job.priority {"priority":16}
8.784s: job.alloc {"annotations":{"sched":{"resource_summary":"rank0/core0"}}}
8.789s: job.start
...

The prototype simply adds attributes.system.begin-time, but from @SteVwonder's earlier comment:

If we go that route, then maybe we add time as a valid type and URI scheme to RFC 26? It would also minimize the changes we need to make to flux mini since we can piggyback off of the dependency specification (once that is in place).

I can't remember if we've updated our thinking here, but it would be nice to keep the "simple" dependencies in flux-core separate from the fully functional directed graph dependency system of flux-depend. Though maybe choosing the right type and URI scheme would be sufficient? I'm not really sure how to fit that in to the RFC though.

Edit: I'm also unsure if the resulting --depend option would be very user friendly. However, for this case, a --begin-time option could easily translate a simpler argument to the required RFC 26 schema.

@SteVwonder, any thoughts?

[SteVwonder]

I'm also unsure if the resulting --depend option would be very user friendly. However, for this case, a --begin-time option could easily translate a simpler argument to the required RFC 26 schema.

Yeah, that sounds like a good way to go.

I can't remember if we've updated our thinking here, but it would be nice to keep the "simple" dependencies in flux-core separate from the fully functional directed graph dependency system of flux-depend. Though maybe choosing the right type and URI scheme would be sufficient? I'm not really sure how to fit that in to the RFC though.

Yeah, that RFC definitely needs some work. While hacking on flux-depend, I realized that the type and scope options really only apply to the string scheme. For the fluid scheme, it doesn't make any sense. Same for a potential time scheme, it would need it's own set of options, probably. So the RFC can definitely be structured to make more sense.

My intial thought is to do whatever makes sense for now and we can deal with RFC later. But for the sake of starting that conversation, I'll post a strawman in another top-level comment of this conversation.

[grondo]

Not to get too far OT, but I like the idea of extending dependency support by adding new schemes. However, what I guess is the URI-based terminology is a little confusing since it introduces a type which is not a type of dependency in the common parlance but a depend-type from the OpenMP specification.

Maybe the type could be renamed if we're editing the RFC anyway.

The list of supported schemes would be extensible by plugins. Each plugin would handle one or more schemes, e.g. flux-core builtin plugins would support begin-time and jobid (or maybe active-jobid). The flux-depend module would support string and fluid schemes. A custom plugin could extend this even further.

Eventually plugins that support dependencies would have to register their supported schemes. This would ensure no conflicts and also would allow the job-manager to ensure all dependencies were "consumed" and issue a useful error if not (e.g. if a "string" dependency were submitted without the flux-depend module/plugin loaded)
hh

[grondo]

Eventually plugins that support dependencies would have to register their supported schemes.

Ah, maybe the way to handle this is that scheme-based dependency plugins register for a job.dependency.<scheme> callback instead of the generic job.state.dependency topic. The job-manager will iterate RFC 26 dependencies calling the scheme callback for each one. The job is rejected if no job.dependency.<scheme> callback is currently registered with an error like "Don't know how to handle dependency scheme foo".

[SteVwonder]

@grondo: Thoughts on this strawman URI scheme: time:<ISO 8601 Timestamp> (e.g., time:2021-05-12T20:42:26Z?) My initial thought was to use a Unix epoch time, but if we are going to list these dependencies in flux jobs, it seemed better to go with a more human readable format. I was also thinking we could get fancy with it and support deadline scheduling, but it seems to break our job state model to have a job cancelled by the depend module after it has already entered the queued state.

For the --begin-time argument, seems like we could support a similar set of values as slurm, with the exception that the "time delta" would be an FSD as opposed to Slurm's format:

   --begin-time=16:00
   --begin-time=now+1h
   --begin-time=now+60s
   --begin-time=2010-01-20T12:34:00

[grondo]

My initial thought was to use a Unix epoch time, but if we are going to list these dependencies in flux jobs, it seemed better to go with a more human readable format.

I think I would prefer storing as Unix epoch (optionally as a float). This removes the need to convert timezones. We can convert the begin-time to any format quite easily in the JobInfo class and thus the human readability isn't an issue I assume. (In fact it would be nice to present an offset instead of absolute time as well and epoch will make this easy)

My next question is where do the dependencies go in the jobspec? attributes.system.dependencies is an array of RFC 26 dependency objects?

For the --begin-time argument, seems like we could support a similar set of values as slurm, with the exception that the "time delta" would be an FSD as opposed to Slurm's format

Don't forget --begin-time=teatime and --begin-time=fika. We can add --begin-time=fluxcoffeehour

[SteVwonder]

My next question is where do the dependencies go in the jobspec? attributes.system.dependencies is an array of RFC 26 dependency objects?

I don't think have a space for it laid out in JobspecV1, but in V2, that (i.e., attributes.system.dependencies) is where it is planned to go: https://github.com/flux-framework/rfc/pull/229/files#diff-27f2d23614014adf43c02e7e852dc2a47344bf64a62a3b12d8b73330c921ebc9R107

Don't forget --begin-time=teatime and --begin-time=fika. We can add --begin-time=fluxcoffeehour

😆

[grondo]

(i.e., attributes.system.dependencies) is where it is planned to go:

Thanks, I found that after my initial comment. We should just go ahead and add this to V1 (formally or informally), since we'll need to support these simple dependencies before v2 is deployed I think

[grondo]

A similar option to Slurm's --begin=TIME has been proposed in #3704. The new flux-mini --begin-time option works by creating a dependency with the begin-time scheme in the generated jobspec. See RFC 26 for a description of the begin-time dependency scheme.