Error on Flux of Helm release and Image Repository

[praveenavasoft]

Describe the bug:

I am trying to automate the CD by using flux with toolkit components and facing two errors.

1. In Helm controller, Helm release is reconciled, the GCP Artifact Registry it is facing an error of “failed to authorize: failed to fetch anonymous token: unexpected status from GET request” and “failed to pull&service” for OCI repositories, the credentials are provided as a secret reference with .dockercofig authentication affiliated with default namespace.
2. In Image Automation Controller, Image repository is reconciling the GCP Artifact Registry it is facing an error of “pull&service=: UNAUTHORIZED: authentication failed” for OCI repositories, the credentials are provided as a secret reference with .dockercofig authentication affiliated with default namespace.

D:\xxx\xxx\xxx>flux get all

NAME             REVISION        SUSPENDED    READY      MESSAGE
xxxxxxxxxx                         False      True     stored artifact for revision 'main@sha1:12345'

NAME           REVISION   SUSPENDED   READY     MESSAGE
xxxxxxxxxx                  False     True    Helm repository is ready

NAME           REVISION     SUSPENDED     READY        MESSAGE
xxxxxxxxxx                    False       False      chart pull error: failed to download chart for remote reference: 
failed to get 'oci://us-east4-docker.pkg.dev/xxx/xxxxx/xxxxx/xxxxxx:0.1.0': failed to authorize: 
failed to fetch anonymous token: unexpected status from GET request to https://us-east4-docker.pkg.dev/v2/token?scope=repository
%!!(MISSING)A(MISSING)xxxxxxxxxxxxxxx%!!(MISSING)F(MISSING)xxxxxx%!!(MISSING)F(MISSING)xxxxxx%!!(MISSING)F(MISSING)xxxxxxx%!
!(MISSING)A(MISSING)pull&service=us-east4-docker.pkg.dev: 403 Forbidden

NAME           LAST SCAN    SUSPENDED     READY       MESSAGE
xxxxxxxxxx                    False       False     scan failed: GET https://us-east4-docker.pkg.dev/v2/token?scope=repository%!A(MISSING)xxxxxxxxx%!F(MISSING)xxxxxxxxx%!F(MISSING)xxxxxxxxxx%!A(MISSING)pull&service=: UNAUTHORIZED: authentication failed

NAME           LATEST IMAGE    READY        MESSAGE
xxxxxxxxxx                     False      referenced ImageRepository has not been scanned yet

NAME             LAST RUN                     SUSPENDED       READY        MESSAGE
xxxxxxxxxx     2023-12-07T17:38:23+05:30        False          True       no updates made

NAME            REVISION   SUSPENDED   READY       MESSAGE
xxxxxxxxxx                   False     False     HelmChart 'flux-system/flux-system-xxxxxxx' is not ready

NAME             REVISION         SUSPENDED    READY         MESSAGE
xxxxxxxxxx                          False      True       Applied revision: main@sha1:12345

Steps to reproduce:

Install Flux.

choco install flux

Flux bootstrap a github repository:

flux bootstrap github --token-auth --owner=abc --repository=abc --branch=main --path=apps --components-extra=image-reflector-controller,image-automation-controller

Flux Helm controller: Helm repo and Helm Release.

flux create source helm abc --url=oci://us-east4-docker.pkg.dev/xxx/xxxxx/xxxx --secret-ref=abc --namespace=abc --export > ./abc.yaml

flux create hr abc --interval=1m0s --source=HelmRepository/abc --chart=abc --chart-version=">0.1.0" --export > ./abc.yaml

Flux Image controller:

flux create image repository abc  - -secret-ref abc - -image xxx/xxx/xxx - -interval 5m

flux create image policy abc - -image-ref abc - - secret-semver 1.0.x - -export > ./abc.yml

flux create image update abc  - -interval 10m - -git-repo-ref abc - - git-repo-path *./xxx/xxx * - -author-name abc - -author-email abc@xxx.com - - checkout-branch main - -push-branch main  - -export > ./abc.yml

Expected behaviour:

Helm Release:

D:\xxx\xxx\xxx>flux get helmrelease -n flux-system
NAME   REVISION  SUSPENDED  READY     MESSAGE
abc                False     True   HelmChart 'flux-system/flux-system-abc' is ready

Image Repository:

D:\xxx\xxx\xxx>kubectl get imagerepository -n abc
NAME      LAST SCAN         TAGS
abc   2023-12-1T22:34:05Z    123

Image Policy

D:\xxx\xxx\xxx>kubectl get imagepolicy -n abc
NAME      LATESTIMAGE
abc   xxx/xxx/xxx:1.0.0

Screenshots and Recordings:

No Reponses

OS/Distro:

Windows cmd/gcloud SDK

Flux version:

flux version 2.1.2

Flux check:

D:\xxx\xxx\xxx>flux check

► checking prerequisites
✔ Kubernetes 1.27.4-gke.900 >=1.25.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.36.2
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.36.1
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.30.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.1.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.1.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.1.2
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta2
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta2
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider:

GitHub

Container Registry Provider:

GCP Artifact Registry

Additional context:

No Responses

[whatnick]

I can report similar error when logging into GHCR Helm OCI repository via dockerconfig tagged secret.