Providing custom CRD to kustomize-controller

[stoetti]

We are using our own CRDs for deploying our applications within k8s. As we evaluate using flux2 for CD we came with the issue to use kustomization on those custom resources. The Kustomize-doc states that the CRD has to be provided withing the configuration (https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/crds/).

From understanding this means that the CRD has to be pushed to each repo where a manifest based on that CRD is used. I searchd the documentation and issues but didn't find a possiblity to provide the CRDs globally to the kustomize-controller or any other flux-component.

Is there a way?
If not, how could this be solved without copying (and maintaining) the CRD in each multiple repos?

[stefanprodan]

Flux reconciles its own custom resources without specifying the CRDs.

[stoetti]

well the kustomization controller throws the following error

{"level":"error","ts":"2021-04-19T07:32:26.821Z","logger":"controller.kustomization","msg":"Reconciliation failed after 41.058422ms, next try in 1m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"data2go-dev","namespace":"ops","revision":"master/c9e979889a60510d3d7831ed4e6dbc6ae24ab192","error":"kustomize build failed: accumulating resources: 2 errors occurred:\n\t* accumulateFile error: \"accumulating resources from '../base/apps': read /tmp/data2go-dev307117718/base/apps: is a directory\"\n\t* accumulateDirector error: \"recursed accumulation of path '/tmp/data2go-dev307117718/base/apps': **Unexpected kind: unknown**\"\n\n","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99"}

[stefanprodan]

Hmm I wander why this works then https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/apps/production/kustomization.yaml#L5 Maybe kustomize needs to know the CRD when using wildcards, if you use strategic merge like the in above example, does it work for you?

[stoetti]

my mistakte the patch needs to be provided differently in order to work

patch: |-  
  op: replace
  path: spec/customField
  value: kustomizedValue

But now I get a validation error, from my understanding of the documentation I need to set "validation=false" on the Kustomization-resource of flux but that does not change anything, the validation error is still thrown.

{"level":"error","ts":"2021-04-19T07:49:44.004Z","logger":"controller.kustomization","msg":"Reconciliation failed after 456.218075ms, next try in 1m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"data2go-dev","namespace":"ops","revision":"master/355d8731bfb2ee3fa24dbc46592339d9a9a160a7","error":"validation failed: error: error validating \"e594992c-4752-4514-931e-c0e4450456b4.yaml\": error validating data: ValidationError(App): unknown field \"postgresInstanceName\" in com.unisoftwareplus.platform.v1alpha2.App; if you choose to ignore these errors, turn validation off with --validate=false\n","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99"}

[stefanprodan]

The validation tells you that there is no postgresInstanceName field declared in your CRD... there is nothing flux can do, you need to use the right fields.

[stoetti]

okay I forgot the initial '/' in the path specification, now it works.

Thanks for the support.