I would like to limit the number of jobs with a given label that can run simultaneously.

[mezhaka]

Here's a scenario: I keep on submitting jobs with some label. Say I have submitted N+K jobs, but only up to N jobs would be able to create pods and transition to Running state, while the rest K will sit till some of the first N jobs finish. In other words only up to N jobs that match some label filter may run simultaneously.

1. Is Kopf something that can help me implement an operator that can limit the amount of jobs running simultaneously?
2. If 1) is true, where do I start to dig?

[thismatters]

What mechanism for submitting jobs do you have in mind?

[nolar]

Hello. Thanks for asking. I have no idea how this could be implemented the best way. Maybe, I would take a look at Kubernetes's built-in affinity & anti-affinity tools for this, or at Kubernetes's system resource allocation (if it can create pseudo-resources, not only CPU/RAM/GPU).

Or maybe even looked at the task itself from a different angle and tried to redesign the whole solution. It looks like the XY Problem here: you are looking for a technical solution Y as presumably the only solution for the original unspoken problem X, while there can be other better solutions.

Kopf can probably solve this task too, as long as you manipulate spec.suspend: true/false field from the operator. Use the Kopf indexes to keep the track of all existing jobs with that label.

import kopf

@kopf.index('jobs', labels={'marker': 'xyz'}, field='spec.suspend', value=True)
def xyz_jobs(namespace, name, **_):
    return {(namespace, name): None}

def has_just_finished(old, new, **_):
    return False  # TODO: somehow, detect if it was done

@kopf.on.update('jobs', when=has_just_finished)
def job_finished(xyz_jobs, **_):
    all_pending_jobs = set(xyz_jobs)
    next_job_ns, next_job_name = random.choice(all_pending_jobs)
    patch_job(next_job_ns, next_job_name, {'spec': {'suspend': False}})

This, however, looks like an overcomplication. And definitely, it is not finished — just the overall idea.

[bobh66]

Kyverno can implement this check fairly easily using poiicies

[mchestnut91]

@mezhaka did you find a solution for this? I have the exact same use-case, so I'm wondering if/how you were able to achieve limiting these jobs?