-
Notifications
You must be signed in to change notification settings - Fork 230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JobSet fails to suspend if nodeSelectors are added on admission #2691
Comments
/assign |
I think that this is a little big deal for the GPU workload since some users inject the tolerations for the GPU Node via resourceFlavor. By this limitation, we can not preempt or deactivate the Jobs with tolerations inserted by the ResourceFlavor, right? |
To workaround this problem we have implemented a custom stop in Kueue, which takes 3 requests to stop:
So, while it is not very performant it works. So it is a high priority, but I would not call it urgent. We could follow a similar pattern for JobSet (one request to suspend, and one to restore PodTemplate), but I hope we can fix it in JobSet itself. |
I think it didn't surface until now because JobSet is mostly used with DWS ProvisioningRequests, where the quota is set very high, so evictions are rare. |
Yeah, that is the reason why we introduced the |
One thing is preemption or deactivation throughput, this 3 step stop mechanism obviously would bring us lower throughput preemption. |
IIUC kubernetes/kubernetes#113221 is about it - to give us an "official" API to do what we do in Kueue. The fact that Kueue sets I think the idea "We could implement mutable scheduling directives after suspension if we set .status.startTime to nil when when we are confident that there all the running pods are at least terminating. " is that |
I agree with you. Additionally, nice to have verifications if the |
What happened:
JobSet fails to suspend if there are nodeSelectors added on admission. It fails because JobSet webhook rejects the update.
What you expected to happen:
JobSet can be suspended and the PodTemplate is restored.
How to reproduce it (as minimally and precisely as possible):
k edit clusterqueue/cluster-queue
and set thestopPolicy: HoldAndDrain
Issue: This should result in stopping the JobSet. However, it continues to run and be admitted. The logs for Kueue show that the mutation request is rejected by JobSet webhook.
The issue is reproduced by the e2e test in the WIP PR: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/2700/pull-kueue-test-e2e-main-1-29/1816503837968568320
Anything else we need to know?:
I fill open a dedicated issue in JobSet and see if we can fix it there. We may need a work-around in Kueue to do 2-step stop:
suspend first, then restore the PodTemplate, but I prefer to avoid it. This is something we aim to avoid for Job in the long run too: kubernetes/kubernetes#113221
Environment:
kubectl version
):git describe --tags --dirty --always
):cat /etc/os-release
):uname -a
):The text was updated successfully, but these errors were encountered: