Wrong placement of the CRDs in the operator chart

[rajewluk]

Testkube operator CRDs are placed in the templates folder of the operator helm chart. In consequence, when testkube fails and webhook service is not available the CRD and its CRs cannot be deleted. Basically, it makes Flux/ArgoCD tools work badly with testkube since they try to delete CRD when CRs are not deleted and that gets stuck if testkube services are already not available (as they get deleted in the random order along with other types of resources). To avoid it, CRDs should be placed in the crds folder of helm chart, as helm documentation states.

[vsukhin]

hey @rajewluk we started with this approach. it was a nice thing, but the prroblem was Helm didn't update CRD witj latest changes in this case. Might be, Helm was improved since that time, for @ypoplavs and @dejanzele t check

[rajewluk]

Understood, but the current approach makes the use of testkube in gitops approach very difficult and unreliable on upgrade.

[rajewluk]

Possibly, making them optional in the templates folder (flag in the values) and having them in the crds folder could help to solve it. First, they would get installed from crds folder, then would get updated from templates (if enabled, the helm app case). In ArgoCD the templates could be basically disabled as those from crds would get synced always by ArgoCD.

[vsukhin]

yes, let's review this issue again and see if we can do it better @ypoplavs

[devantler]

Any progress on this? This seems to still be the case when using Flux GitOps:

Helm install failed for release testkube/testkube with chart testkube@2.0.6: failed post-install: unable to build kubernetes object for deleting hook testkube/charts/testkube-api/templates/testworkflows/builtin-templates/distribute.yaml: resource mapping not found for name: "distribute--evenly" namespace: "" from "": no matches for kind "TestWorkflowTemplate" in version "testworkflows.testkube.io/v1"
ensure CRDs are installed first

[vsukhin]

hey, @devantler it should be realted to test workkflow template post hook installation, which should happen after CRD one. @tkonieczny and @emil2k can clarify

[devantler]

Awesome, looking forward to hear from them. And just for context, I am currently not setting any Helm values:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: testkube
spec:
  interval: 1m
  chart:
    spec:
      chart: testkube
      version: 2.0.6
      sourceRef:
        kind: HelmRepository
        name: testkube
  # https://github.com/kubeshop/helm-charts/blob/develop/charts/testkube/values.yaml
  values: {}

[devantler]

I made it work for now by deploying the CRDs prior to deploying the Helm chart.

# Temporary solution!!!
# This Flux Kustomization deploys the CRDs for the Testkube Helm chart.
# The CRDs are copied from https://github.com/kubeshop/testkube-operator/tree/main/config/crd/bases
# For them to work you need to add the following labels and annotations to the CRDs:
#
# ```yaml
# annotations:
#   meta.helm.sh/release-name: testkube
#   meta.helm.sh/release-namespace: testkube
# labels:
#   app.kubernetes.io/managed-by: Helm
# ```
#
# And you also need to ensure the testkube.yaml Flux Kustomization depends on this one.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: testkube-crds
  namespace: flux-system
  labels:
    app.kubernetes.io/sops: enabled
    app.kubernetes.io/post-build-variables: enabled
spec:
  interval: 1m
  targetNamespace: testkube
  sourceRef:
    kind: OCIRepository
    name: flux-system
  path: testkube/crds
  prune: true
  wait: true

[vsukhin]

thank you @devantler we didn't have time to investigate it so far