This is a sample repo demonstrating how to create a custom scheduler in Kubernetes.
For this proof-of-concept we just add a custom scheduler plugin that will schedule pods only on nodes which match a given label.
Obviously, it would not make sense to use this plugin in reality, since the same behaviour can easily be achieved using built-in functionality, such as using a nodeSelector in your deployment.
If we wanted more control over how our pods get scheduled however, it would be reasonably straightforward to modify this plugin to add the desired behaviour.
See demo/README.md for instructions on how to try out this custom scheduler in a local minikube deployment.
While there are several resources in the Kubernetes documentation regarding how the scheduler works and how it can be extended, it was not clear to me how to go about actually implementing and deploying a custom scheduler.
At first, it appeared as though you can just write your own custom scheduler, but then I discovered there was a better way, using the Scheduling Framework. As mentioned in the docs, The scheduling framework is a pluggable architecture for the Kubernetes scheduler. The APIs allow most scheduling features to be implemented as plugins, while keeping the scheduling "core" lightweight and maintainable. This means we can just write a custom plugin for the specific point of the scheduling cycle we're interested in.
Although it appears fairly straightforward to write your own scheduler plugin, being neither an expert in Kubernetes or Go, I ran into a few issues along the way (such as fixing Go dependencies, and what configuration to use when deploying my plugin). Therefore, perhaps this repo will be useful to others who want to implement their own custom scheduler logic.
- This repo for a custom scheduler written by Chris Seto at Cockroach Labs was very useful as a reference
- The Kubernetes Scheduling Framework
- Source code for other scheduler plugins