kustomize-controller: ignored health checks

[seaneagan]

It may be the case sometimes that one of your health checks is failing, and you have identified that the issue is low impact, so you don't want the health check to stop the progress of the deployment, so you want the Ready condition to not be marked False because of the failure. However, rather than turning off the health check you may want to leave it for observability in order to debug the failure going forward. In this case it would be useful to mark the health check as ignored. This would be similar to the helm test ignore functionality in the helm-controller. It would be useful to separately configure whether to ignore Failed vs InProgress status.

API proposal:

  healthChecks:
    - kind: Deployment
      name: healthy
    - kind: Deployment
      name: unhealthy
      ignoreFailed: true
    - kind: Deployment
      name: unhealthy
      ignoreInProgress: true

Then all health checks (including ignored ones) can be exposed via a new status condition e.g. Healthy, while the Ready condition will only be False upon non-ignored health check failures. The Healthy condition could potentially still be kept up to date even if the kustomization is suspended.

If #324 is implemented, there would need to be a way to specify the default health checks for resources not explicitly included in the list. For example:

Check all resources, ignore specific failure:

  healthChecks:
    default:
      enable: true # default true as proposed in #324
    custom:
      - kind: Deployment
        name: unhealthy
        ignoreFailed: true

Wait for a single resource:

  healthChecks:
    default:
      enable: false
    custom:
      - kind: Deployment
        name: healthy

Ignore all failed and in progress:

  healthChecks:
    default:
      ignoreFailed: true
      ignoreInProgress: true

[vladimirfx]

Explicit health checks raise encapsulation issue , IMO. It is establish strong coupling between kustomization content and it gotk activation resource Kustomization.
Our Ops team do not know all resources deployed by kustomization and should not do.
Agree with wide ignorance mechanism. And propose some fine grained ignore list for adhoc ignorance of readiness of certain resources. It is coupling too, but by far less destructive. Moreover absence of ignored resources should log event and proceed processing.