Failed to create storage handle using createStorageHandle

[vparmeland]

Describe the issue
Sometimes my pods remain in error on creation
toto-xxxxxx-d6z6b 3/4 CreateContainerError 1 (51m ago) 54m
I have to kill the pod for it to start normally

System & Version:

• Platform [GKE 1.29]
• Version [FUSE CSI driver sidecar 1.4.3]

Steps to reproduce the behavior with following information:

 ---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redacted-pv
spec:
  accessModes:
  - ReadOnlyMany
  capacity:
    storage: 500Gi
  storageClassName: toto-storage-class
  claimRef:
    namespace: {{ .Values.namespace }}
    name: redacted-pv
  mountOptions:
    - implicit-dirs
    - uid=1001
    - gid=3003
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeHandle: "{{ .Values.global.gcpProject }}-toto-files"
    volumeAttributes:
      gcsfuseLoggingSeverity: warning  
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redacted-pv
  namespace: {{ .Values.namespace }}
spec:
  accessModes:
  - ReadOnlyMany
  resources:
    requests:
      storage: 500Gi
  volumeName: redacted-pv
  storageClassName: toto-storage-class
---

12:52:22.342681       1 main.go:48] Running Google Cloud Storage FUSE CSI driver sidecar mounter version v1.4.3-gke.8 
12:52:23.845214       1 sidecar_mounter_config.go:101] connecting to socket "/gcsfuse-tmp/.volumes/redacted-pv/socket"
12:52:23.848660       1 fdchannel.go:48] get the underlying socket 
12:52:23.848701       1 fdchannel.go:60] calling recvmsg... 
12:52:23.912438       1 fdchannel.go:69] parsing SCM...     
12:52:23.912467       1 fdchannel.go:76] parsing SCM_RIGHTS...  
12:52:23.912821       1 sidecar_mounter_config.go:269] gcsfuse config file content: map[cache-dir: logging:map[file-path:/dev/fd/1 format:json severity:warning]] 
12:52:23.928456       1 sidecar_mounter.go:51] start to mount bucket "gcp-bucket-toto-file" for volume "redacted-pv"  
12:52:23.929020       1 sidecar_mounter.go:70] gcsfuse mounting with args [--app-name gke-gcs-fuse-csi --foreground --uid 1001 --gid 3003 --temp-dir /gcsfuse-buffer/.volumes/redacted-pv/temp-dir --config-file /gcsfuse-tmp/.volumes/redacted-pv/config.yaml --implici 
12:52:23.929503       1 main.go:73] waiting for SIGTERM signal... 
12:52:24.056520       1 sidecar_mounter.go:105] gcsfuse for bucket "gcp-bucket-toto-file", volume "redacted-pv" started with process id 14  
{"timestamp":{"seconds":1726577548,"nanos":656330481},"severity":"ERROR","message":"Error while mounting gcsfuse: Failed to create storage handle using createStorageHandle: go storage client creation failed: while creating http endpoint: while fetching tokenSource: DefaultTokenSource: google: could not find def │
Failed to create storage handle using createStorageHandle: go storage client creation failed: while creating http endpoint: while fetching tokenSource: DefaultTokenSource: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information  
12:52:28.664260       1 logger.go:60] gcsfuse exited with error: exit status 1  
14:12:55.327789       1 main.go:110] received SIGTERM signal, waiting for all the gcsfuse processes exit... 
14:12:55.327925       1 main.go:118] exiting sidecar mounter... 
 Stream closed EOF for namespacea/toto-xxxxx-d6z6b (gke-gcsfuse-sidecar)

Thanks

[raj-prince]

Hi @vparmeland,

The failure is related to auth configuration mechanism in GKE for gcsfuse-csi-driver. GCSFuse is behaving as expected, it requires application default credential to mount GCS bucket which is not there is your case, hence the failure.

https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver is the better place to create the issue.

Overall guide to access GCS bucket with gcsfuse CSI driver - https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver

Specifically GKE authentication using Workload Identity Federation: https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#authentication