diff --git a/content/en/blog/_posts/2022-09-29-immutability-with-cel.md b/content/en/blog/_posts/2022-09-29-immutability-with-cel.md new file mode 100644 index 0000000000000..c857b06a1fbc3 --- /dev/null +++ b/content/en/blog/_posts/2022-09-29-immutability-with-cel.md @@ -0,0 +1,655 @@ +--- +layout: blog +title: "Enforce CRD Immutability with CEL Transition Rules" +date: 2022-09-29 +slug: enforce-immutability-using-cel +--- + +**Author:** [Alexander Zielenski](https://github.com/alexzielenski) (Google) + +Immutable fields can be found in a few places in the built-in Kubernetes types. +For example, you can't change the `.metadata.name` of an object. Specific objects +have fields where changes to existing objects are constrained; for example, the +`.spec.selector` of a Deployment. + +Aside from simple immutability, there are other common design patterns such as +lists which are append-only, or a map with mutable values and immutable keys. + +Until recently the best way to restrict field mutability for CustomResourceDefinitions +has been to create a validating +[admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/#what-are-admission-webhooks): +this means a lot of complexity for the common case of making a field immutable. + +Beta since Kubernetes 1.25, CEL Validation Rules allow CRD authors to express +validation constraints on their fields using a rich expression language, +[CEL](https://github.com/google/cel-spec). This article explores how you can +use validation rules to implement a few common immutability patterns directly in +the manifest for a CRD. + +## Basics of validation rules + +The new support for CEL validation rules in Kubernetes allows CRD authors to add +complicated admission logic for their resources without writing any code! + +For example, A CEL rule to constrain a field `maximumSize` to be greater than a +`minimumSize` for a CRD might look like the following: + +```yaml +rule: | + self.maximumSize > self.minimumSize +message: 'Maximum size must be greater than minimum size.' +``` + +The rule field contains an expression written in CEL. `self` is a special keyword +in CEL which refers to the object whose type contains the rule. + +The message field is an error message which will be sent to Kubernetes clients +whenever this particular rule is not satisfied. + +For more details about the capabilities and limitations of Validation Rules using +CEL, please refer to +[validation rules](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules). +The [CEL specification](https://github.com/google/cel-spec) is also a good +reference for information specifically related to the language. + +## Immutability patterns with CEL validation rules +This section implements several common use cases for immutability in Kubernetes +CustomResourceDefinitions, using validation rules expressed as +[kubebuilder marker comments](https://book.kubebuilder.io/reference/markers/crd.html). +Resultant OpenAPI generated by the kubebuilder marker comments will also be +included so that if you are writing your CRD manifests by hand you can still +follow along. + +## Project setup + +To use CEL rules with kubebuilder comments, you first need to set up a Golang +project structure with the CRD defined in Go. + +You may skip this step if you are not using kubebuilder or are only interested +in the resultant OpenAPI extensions. + +Begin with a folder structure of a Go module set up like the following. If +you have your own project already set up feel free to adapt this tutorial to your liking: + +{{< mermaid >}} +graph LR + . --> generate.go + . --> pkg --> apis --> stable.example.com --> v1 + v1 --> doc.go + v1 --> types.go + . --> tools.go +{{}} + +This is the typical folder structure used by Kubernetes projects for defining new API resources. + +`doc.go` contains package-level metadata such as the group and the version: +```go +// +groupName=stable.example.com +// +versionName=v1 +package v1 +``` + +`types.go` contains all type definitions in stable.example.com/v1 + +```go +package v1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// An empty CRD as an example of defining a type using controller tools +// +kubebuilder:storageversion +// +kubebuilder:subresource:status +type TestCRD struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec TestCRDSpec `json:"spec,omitempty"` + Status TestCRDStatus `json:"status,omitempty"` +} + +type TestCRDStatus struct {} +type TestCRDSpec struct { + // You will fill this in as you go along +} +``` + +`tools.go` contains a dependency on [controller-gen](https://book.kubebuilder.io/reference/generating-crd.html#generating-crds) which will be used to generate the CRD definition: + +```go +//go:build tools + +package celimmutabilitytutorial + +// Force direct dependency on code-generator so that it may be executed with go run +import ( + _ "sigs.k8s.io/controller-tools/cmd/controller-gen" +) +``` + +Finally, `generate.go`contains a `go:generate` directive to make use of +`controller-gen`. `controller-gen` parses our `types.go` and creates generates +CRD yaml files into a `crd` folder: + +```go +package celimmutabilitytutorial + +//go:generate go run sigs.k8s.io/controller-tools/cmd/controller-gen crd paths=./pkg/apis/... output:dir=./crds +``` + + +You may now want to add dependencies for our definitions and test the code generation: + +```shell +cd cel-immutability-tutorial +go mod init / +go mod tidy +go generate ./... +``` + +After running these commands you now have completed the basic project structure. +Your folder tree should look like the following: + +{{< mermaid >}} +graph LR + . --> crds --> stable.example.com_testcrds.yaml + . --> generate.go + . --> go.mod + . --> go.sum + . --> pkg --> apis --> stable.example.com --> v1 + v1 --> doc.go + v1 --> types.go + . --> tools.go + +{{}} + +The manifest for the example CRD is now available in `crds/stable.example.com_testcrds.yaml`. + +## Immutablility after first modification +A common immutability design pattern is to make the field immutable once it has +been first set. This example will throw a validation error if the field after +changes after being first initialized. + +```go +// +kubebuilder:validation:XValidation:rule="!has(oldSelf.value) || has(self.value)", message="Value is required once set" +type ImmutableSinceFirstWrite struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // +kubebuilder:validation:Optional + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable" + // +kubebuilder:validation:MaxLength=512 + Value string `json:"value"` +} +``` + +The `+kubebuilder` directives in the comments inform controller-gen how to +annotate the generated OpenAPI. The `XValidation` rule causes the rule to appear +among the `x-kubernetes-validations` OpenAPI extension. Kubernetes then +respects the OpenAPI spec to enforce our constraints. + +To enforce a field's immutability after its first write, you need to apply the following constraints: +1. Field must be allowed to be initially unset `+kubebuilder:validation:Optional` +2. Once set, field must not be allowed to be removed: `!has(oldSelf.value) | has(self.value)` (type-scoped rule) +3. Once set, field must not be allowed to change value `self == oldSelf` (field-scoped rule) + +Also note the additional directive `+kubebuilder:validation:MaxLength`. CEL +requires that all strings have attached max length so that it may estimate the +computation cost of the rule. Rules that are too expensive will be rejected. +For more information on CEL cost budgeting, check out the other tutorial. + +### Example usage + +Generating and installing the CRD should succeed: +```shell +# Ensure the CRD yaml is generated by controller-gen +go generate ./... +kubectl apply -f crds/stable.example.com_immutablesincefirstwrites.yaml +``` +```console +customresourcedefinition.apiextensions.k8s.io/immutablesincefirstwrites.stable.example.com created +``` + +Creating initial empty object with no `value` is permitted since `value` is `optional`: +```shell +kubectl apply -f - <: Invalid value: "object": Value is required once set +``` + +### Generated schema +Note that in the generated schema there are two separate rule locations. +One is directly attached to the property `immutable_since_first_write`. +The other rule is associated with the crd type itself. + +```yaml +openAPIV3Schema: + properties: + value: + maxLength: 512 + type: string + x-kubernetes-validations: + - message: Value is immutable + rule: self == oldSelf + type: object + x-kubernetes-validations: + - message: Value is required once set + rule: '!has(oldSelf.value) || has(self.value)' +``` + +## Immutability upon object creation + +A field which is immutable upon creation time is implemented similarly to the +earlier example. The difference is that that field is marked required, and the +type-scoped rule is no longer necessary. + +```go +type ImmutableSinceCreation struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // +kubebuilder:validation:Required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable" + // +kubebuilder:validation:MaxLength=512 + Value string `json:"value"` +} +``` + +This field will be required when the object is created, and after that point will +not be allowed to be modified. Our CEL Validation Rule `self == oldSelf` + +### Usage example + +Generating and installing the CRD should succeed: +```shell +# Ensure the CRD yaml is generated by controller-gen +go generate ./... +kubectl apply -f crds/stable.example.com_immutablesincecreations.yaml +``` +```console +customresourcedefinition.apiextensions.k8s.io/immutablesincecreations.stable.example.com created +``` + +Applying an object without the required field should fail: + +```shell +kubectl apply -f - <: Invalid value: "null": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation +``` + +Now that the field has been added, the operation is permitted: +```shell +kubectl apply -f - <: Invalid value: "null": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation +``` + +### Generated schema +```yaml +openAPIV3Schema: + properties: + value: + maxLength: 512 + type: string + x-kubernetes-validations: + - message: Value is immutable + rule: self == oldSelf + required: + - value + type: object +``` + +## Append-only list of containers +In the case of ephemeral containers on Pods, Kubernetes enforces that the +elements in the list are immutable, and can’t be removed. The following example +shows how you could use CEL to achieve the same behavior. + +```go +// +kubebuilder:validation:XValidation:rule="!has(oldSelf.value) || has(self.value)", message="Value is required once set" +type AppendOnlyList struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=100 + // +kubebuilder:validation:XValidation:rule="oldSelf.all(x, x in self)",message="Values may only be added" + Values []v1.EphemeralContainer `json:"value"` +} +``` + +1. Once set, field must not be deleted: `!has(oldSelf.value) || has(self.value)` (type-scoped) +2. Once a value is added it is not removed: `oldSelf.all(x, x in self)` (field-scoped) +2. Value may be initially unset: `+kubebuilder:validation:Optional` + +Note that for cost-budgeting purposes, `MaxItems` is also required to be specified. + +### Example usage + +Generating and installing the CRD should succeed: +```shell +# Ensure the CRD yaml is generated by controller-gen +go generate ./... +kubectl apply -f crds/stable.example.com_appendonlylists.yaml +``` +```console +customresourcedefinition.apiextensions.k8s.io/appendonlylists.stable.example.com created +``` + +Creating an inital list with one element inside should succeed without problem: +```shell +kubectl apply -f - <: Invalid value: "object": Value is required once set +``` + +### Generated schema +```yaml +openAPIV3Schema: + properties: + value: + items: ... + maxItems: 100 + type: array + x-kubernetes-validations: + - message: Values may only be added + rule: oldSelf.all(x, x in self) + type: object + x-kubernetes-validations: + - message: Value is required once set + rule: '!has(oldSelf.value) || has(self.value)' +``` + +## Map with append-only keys, immutable values + +```go +// A map which does not allow keys to be removed or their values changed once set. New keys may be added, however. +// +kubebuilder:validation:XValidation:rule="!has(oldSelf.values) || has(self.values)", message="Value is required once set" +type MapAppendOnlyKeys struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxProperties=10 + // +kubebuilder:validation:XValidation:rule="oldSelf.all(key, key in self && self[key] == oldSelf[key])",message="Keys may not be removed and their values must stay the same" + Values map[string]string `json:"values,omitempty"` +} +``` + +1. Once set, field must not be deleted: `!has(oldSelf.values) || has(self.values)` (type-scoped) +2. Once a key is added it is not removed nor is its value modified: `oldSelf.all(key, key in self && self[key] == oldSelf[key])` (field-scoped) +3. Value may be initially unset: `+kubebuilder:validation:Optional` + +### Example usage +Generating and installing the CRD should succeed: +```shell +# Ensure the CRD yaml is generated by controller-gen +go generate ./... +kubectl apply -f crds/stable.example.com_mapappendonlykeys.yaml +``` +```console +customresourcedefinition.apiextensions.k8s.io/mapappendonlykeys.stable.example.com created +``` + +Creating an initial object with one key within `values` should be permitted: +```shell +kubectl apply -f - <: Invalid value: "object": Value is required once set +``` + +### Generated schema + +```yaml +openAPIV3Schema: + description: A map which does not allow keys to be removed or their values + changed once set. New keys may be added, however. + properties: + values: + additionalProperties: + type: string + maxProperties: 10 + type: object + x-kubernetes-validations: + - message: Keys may not be removed and their values must stay the same + rule: oldSelf.all(key, key in self && self[key] == oldSelf[key]) + type: object + x-kubernetes-validations: + - message: Value is required once set + rule: '!has(oldSelf.values) || has(self.values)' +``` + +# Going further +The above examples showed how CEL rules can be added to kubebuilder types. +The same rules can be added directly to OpenAPI if writing a manifest for a CRD by hand. + +For native types, the same behavior can be achieved using kube-openapi’s marker +[`+validations`](https://github.com/kubernetes/kube-openapi/blob/923526ac052c59656d41710b45bbcb03748aa9d6/pkg/generators/extension.go#L69). + +Usage of CEL within Kubernetes Validation Rules is so much more powerful than +what has been shown in this article. For more information please check out +[validation rules](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules) +in the Kubernetes documentation and [CRD Validation Rules Beta](https://kubernetes.io/blog/2022/09/23/crd-validation-rules-beta/) blog post.