Controller keeps scheduling Pods from a cancelled Build

[artem-zinnatullin]

Steps to reproduce:

1. Set up buildkite/agent-stack-k8s with max-in-flight: 2
2. Push a new commit to an open PR which, it triggers 10 new jobs, controller schedules 2 Pods at a time respecting max-in-flight: 2
3. Cancel the build in Buildkite UI / API, all 10 new jobs display as Cancelled in Buildkite UI
4. Observe controller still scheduling Pods for that build flooding the K8S cluster (kubectl describe pod shows BUILDKITE_COMMIT which is cancelled) and preventing other Jobs from being handled

Environment:

• AWS EKS v1.25
• buildkite/agent-stack-k8s v0.15.0

[DrJosh9000]

Thanks for the report @artem-zinnatullin, I can see how this happens.