Interaction with other Kubernetes projects

[severinson]

Armada is designed for high-throughput scheduling of pods across several worker clusters. To achieve this, Armada relies on a multi-stage scheduling strategy, where

1. pods are submitted to the Armada server, which stores queues of pods in a high-throughput storage layer,
2. the Armada scheduler assigns pods to Kubernetes clusters, and, finally,
3. the in-cluster scheduler (e.g., kube-scheduler) assigns pods to nodes.

This strategy ensures that Armada has a global view of all pods in the system, which is necessary to fairly balance resource usage between users. Further, it ensures that a large number of pods can be ingested over a short span of time without overloading etcd. However, this strategy makes it difficult for Armada to interact with other components in the Kubernetes ecosystem that expect to interact directly with the Kubernets API, e.g., to

• create objects other than pods or
• dynamically create and destroy objects.

To address this issue, we propose for Armada to control the scheduling of pods created directly within a Kubernetes worker cluster that Armada manages. Specifically:

1. A pod is created in a Kubernetes cluster managed by Armada, e.g., by a Dask coordinator node requesting more workers. This pod is marked in some way that prevents kube-scheduler from assigning it to a node.
2. The Armada scheduler detects the new pod and adds it to its internal queue.
3. Eventually the Armada scheduler detrmined that the pod should be run, at which point the mark preventing it from being scheduled is removed.
4. Kube-scheduler adds the pods to its internal queue and assigns it to a node.

Using this approach, other components can interact directly with the Kubernetes API while Armada retains the global view of all pods necessary for scheduling across many clusters. Objects other than pods are created immediately without needing to go through the Armada scheduler. Any objects created in this way inherit the queue and job set of the object that created them. Pods created in this way are always scheduled by the Armada scheduler onto the cluster they were created in.

The mark that prevents scheduling could be implemented, e.g., via a kube-scheduler plugin that interacts with Armada.

Open questions:

• How do we handle garbage collection of objects created in this way? Automatically setup owner references? Ephemeral namespaces?
• Do we need to constrain what objects are created? How do we handle auth?
• Are there caveats to this approach? E.g., how gracefully does various services handle pods staying in pending for a long time?

[kannon92]

1. A Pod is created in a Kubernetes cluster managed by Armada.

Maybe this KEP would be useful for this: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/3521-pod-scheduling-readiness/README.md

Its alpha in 1.26.

I like this idea!

How do we handle garbage collection of objects created in this way? Automatically setup owner references? Ephemeral namespaces?

Do we need to constrain what objects are created? How do we handle auth?

I feel like maybe queues should start to include what kind of objects they are allowed to create?
something like: user a has ability to create dask jobs.

Are there caveats to this approach? E.g., how gracefully does various services handle pods staying in pending for a long time?

If objects can be CRDs I think we may need to take this case by case as every single object probbaly has their own way of GC. I think an ephemeral namespace may be a heavy-handed but necessary approach for this. The other option (if crd) is to do the delete of the object but that may require too much support depending on what kind of objects we want to support.

create objects other than pods

This is a scary phase to me. Pretty much everything other than service/ingress ends up being a pod. A user could schedule a deployment object and I think the underlying Pods should be scheduled in Armada. A CRD will most likely create some kind of Pod under the hood. I almost think everything (minus services/ingress) should be handled via the scheduler.