-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sidecar container prevents istio-validation init container from starting in GKE Autopilot cluster #322
Comments
The Can you share the cluster ID with me? You can get the ID by running |
Thanks @songjiaxun |
Checked the Pod creation audit log using the following query:
I see the sidecar
We are seeing similar issues from other users. The cause of this issue is that the native sidecar feature is not recognized by some webhooks, so the incompatible webhook will remove the Workaround 1 A quick workaround is to add a new node pool using gcloud container --project "<your-project>" node-pools create "pool-dummy" --cluster "<your-cluster-name>" --location "<your-cluster-location>" --node-version "1.28" --machine-type "e2-micro" --image-type "COS_CONTAINERD" --disk-type "pd-standard" --disk-size "10" --num-nodes "1" Workaround 2 You can manually inject the apiVersion: v1
kind: Pod
metadata:
name: test
annotations:
# gke-gcsfuse/volumes: "true" <- remove this annotation
spec:
containers:
# add the gke-gcsfuse-sidecar BEFORE your workload container
- args:
- --v=5
image: gke.gcr.io/gcs-fuse-csi-driver-sidecar-mounter:v1.4.2-gke.0@sha256:80c2a52aaa16ee7d9956a4e4afb7442893919300af84ae445ced32ac758c55ad
imagePullPolicy: IfNotPresent
name: gke-gcsfuse-sidecar
resources:
requests:
cpu: 250m
ephemeral-storage: 5Gi
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /gcsfuse-tmp
name: gke-gcsfuse-tmp
- mountPath: /gcsfuse-buffer
name: gke-gcsfuse-buffer
- mountPath: /gcsfuse-cache
name: gke-gcsfuse-cache
- name: your-workload
...
volumes:
# add following three volumes
- emptyDir: {}
name: gke-gcsfuse-tmp
- emptyDir: {}
name: gke-gcsfuse-buffer
- emptyDir: {}
name: gke-gcsfuse-cache Long-term fix We are actively working on fixing this issue. |
Thank you - workaround 2 - inject the |
Hi, I encountered the same issue, and I was able to solve it using workaround 2 mentioned here, thanks. I have a quick question. Could you give me reasons why workaround 1 use v1.28? I understand that the native sidecar container feature was introduced at v1.28 (ref). So we should use more older version for workaround 1 I just thought. |
GKE Autopilot Cluster.
Rapid release channel - Cluster and Node at: 1.30.2-gke.1587003
Deployment is a Knative service.
The Pod does not start with reason "Container istio-proxy is waiting".
Similar issues:
#20
#53
As per https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#pod-annotations the Pod annotations include:
Full Pod spec:
GCSFuse container logs:
The text was updated successfully, but these errors were encountered: