[Flaky] when Creating a multikueue admission check Should run a kubeflow XGBoostJob

[alculquicondor]

What happened:

End To End MultiKueue Suite: kindest/node:v1.30.0: [It] MultiKueue when Creating a multikueue admission check Should run a kubeflow XGBoostJob on worker if admitted expand_less	9s
{Timed out after 5.000s.
The function passed to Eventually failed at /home/prow/go/src/kubernetes-sigs/kueue/test/e2e/multikueue/e2e_test.go:688 with:
Expected object to be comparable, diff:   &v1.ReplicaStatus{
- 	Active:        1,
+ 	Active:        0,
- 	Succeeded:     0,
+ 	Succeeded:     1,
  	Failed:        0,
  	LabelSelector: nil,
  	Selector:      "",
  }
 failed [FAILED] Timed out after 5.000s.
The function passed to Eventually failed at /home/prow/go/src/kubernetes-sigs/kueue/test/e2e/multikueue/e2e_test.go:688 with:
Expected object to be comparable, diff:   &v1.ReplicaStatus{
- 	Active:        1,
+ 	Active:        0,
- 	Succeeded:     0,
+ 	Succeeded:     1,
  	Failed:        0,
  	LabelSelector: nil,
  	Selector:      "",
  }
In [It] at: /home/prow/go/src/kubernetes-sigs/kueue/test/e2e/multikueue/e2e_test.go:703 @ 08/15/24 06:05:55.326
}

What you expected to happen:

Test to pass

How to reproduce it (as minimally and precisely as possible):

• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-kueue-test-multikueue-e2e-main/1823962283705896960
• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-kueue-test-multikueue-e2e-main/1821968118130413568

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Kueue version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[alculquicondor]

/assign @mszadkow

[tenzen-y]

/kind flake

The XGBoostJob has some state transition bugs. So, maybe we need to remove the test case from Kueue or fix the root bug in the training-operator.

[alculquicondor]

I see, thanks for the context.

@mszadkow any chance you can take a look in the training-operator code?
In the meantime, let's disable this test by calling ginkgo.Skip() with an accompanying comment.

[mszadkow]

@tenzen-y Can you explain more about the transition bug, is it known one?

[mszadkow]

Yes, sure I can have a look there but like you said will skip it for now.

[tenzen-y]

@tenzen-y Can you explain more about the transition bug, is it known one?

Depending on historical reasons, we just used to rerun the failed flaky tests in the TrainingOperator.
So, we do not have a dedicated issue for specific transitions.

But, we explained the transition issue a little bit here: kubeflow/training-operator#1711