HelmRelease applies the values.yaml to the chart AFTER the Helm chart is applied

[taimoorgit]

~ ❯ flux version                                                                                                                                                                                                                                                                 Ruby 3.0.0 09:49:35 am
flux: v2.2.3
distribution: flux-v2.2.3
helm-controller: v0.37.4
image-automation-controller: v0.37.1
image-reflector-controller: v0.31.2
kustomize-controller: v1.2.2
notification-controller: v1.2.4
source-controller: v1.2.4

I am trying to deploy a HelmRelease that looks like this:

apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: self-hosted
  namespace: self-hosted
spec:
  interval: 1m0s
  chart:
    spec:
      chart: opslevel
      sourceRef:
        kind: HelmRepository
        name: self-hosted
      valuesFiles:
        - values.yaml
  valuesFrom:
    - kind: Secret
      name: self-hosted-values
      valuesKey: values.yaml

Here's the referenced secret:

~ ❯ kubectl get secret -o yaml self-hosted-values                                                                                                                                                                                                          Ruby 3.0.0 ○ new-dev/self-hosted 10:15:22 am
apiVersion: v1
data:
  values.yaml: <base64 encoded data here>
kind: Secret
metadata:
  creationTimestamp: "2024-06-18T23:05:00Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: self-hosted-values
  namespace: self-hosted
  resourceVersion: "25440839"
  uid: 42c4bf56-3b43-4f28-bae8-04c2638d794e
type: Opaque

This is what I intend to do:

1. My Helm chart I am deploying lets me do things like create Secrets (like one containing MySQL DB password) and set volume sizes used by Deployments by using the values.yaml.
2. The values.yaml I want to use is stored in the self-hosted-values Secret. That Secret contains the modified values.yaml I want to use. Basically I want to do the same thing as helm install opslevel <url> -f values.yaml
3. I am using plain SOPS without any kustomization.yaml for this Secret. However even without using SOPS/flux to manage the Secret and just applying the Secret manually with kubectl apply -f secret-file.yaml I run into the same issue.

Here's what actually happens when I use Flux to apply the HelmRelease:

1. Flux creates all resources (Deployments, volumes, Secrets) but using the default values.yaml and not the one I referenced in valuesFrom: with all of my modified values taken from self-hosted-values.
2. As a result all the volume sizes are way larger than I expected them to be. The Deployment for the MySQL DB was created with the default password and not the one I defined in my custom values.yaml.
3. The pods that connect to the MySQL DB try to use the password from my custom values.yaml. Even the livenessProbe uses the new password and not the one the default, which is what was actually used when the MySQL Deployment and volume was created and so the livenessProbe fails too.
4. When I kubectl get <configmap|secret> -o yaml <name> I can see that everything matches what's in my custom values.yaml, so it seems like my values.yaml was applied after the helm install used the default values.yaml?

P.S. I have tested this by using a plain Helm install without Flux and it worked without encountering this problem.

Flux logs

At 17:30 I pushed a commit that re-adds the HelmRelease.

$ flux logs -n self-hosted --since 6h | sort
2024-06-19T17:30:38.985Z info HelmRelease/self-hosted.self-hosted - Created HelmChart/self-hosted/self-hosted-self-hosted with SourceRef 'HelmRepository/self-hosted/self-hosted'
2024-06-19T17:30:39.003Z info HelmRelease/self-hosted.self-hosted - HelmChart 'self-hosted/self-hosted-self-hosted' is not ready: latest generation of object has not been reconciled
2024-06-19T17:30:42.172Z info HelmChart/self-hosted-self-hosted.self-hosted - pulled 'opslevel' chart with version '2024.6.18'
2024-06-19T17:30:42.200Z info HelmRelease/self-hosted.self-hosted - HelmChart/self-hosted/self-hosted-self-hosted with SourceRef 'HelmRepository/self-hosted/self-hosted' is in-sync
2024-06-19T17:30:42.213Z info HelmRelease/self-hosted.self-hosted - release not installed: no release in storage for object
2024-06-19T17:30:42.243Z info HelmRelease/self-hosted.self-hosted - running 'install' action with timeout of 5m0s
2024-06-19T17:35:33.006Z info HelmChart/self-hosted-self-hosted.self-hosted - artifact up-to-date with remote revision: '2024.6.18'
2024-06-19T17:35:44.861Z info HelmRelease/self-hosted.self-hosted - release is in a failed state
2024-06-19T17:35:44.889Z error HelmRelease/self-hosted.self-hosted - Reconciler error terminal error: exceeded maximum retries: cannot remediate failed release
2024-06-19T17:35:53.432Z info HelmRelease/self-hosted.self-hosted - HelmChart/self-hosted/self-hosted-self-hosted with SourceRef 'HelmRepository/self-hosted/self-hosted' is in-sync
2024-06-19T17:35:53.470Z info HelmRelease/self-hosted.self-hosted - release is in a failed state
2024-06-19T17:35:53.485Z error HelmRelease/self-hosted.self-hosted - Reconciler error terminal error: exceeded maximum retries: cannot remediate failed release
2024-06-19T17:40:22.601Z info HelmChart/self-hosted-self-hosted.self-hosted - artifact up-to-date with remote revision: '2024.6.18'

[taimoorgit]

Closing discussion since it turns out that this is actually a problem with our Helm Chart not dealing with initializing PersistentVolumeClaims in the correct order rather than an issue with anything Flux related.