Race when creating azure-vnet lock directory with permissions

[QxBytes]

What happened:
On the first boot, no CNI binary is on the node, and so k8s creates the /var/run/azure-vnet directory with 0755 permissions automatically because it is a mount part of the azure-cns daemonset. Then the CNI is deployed.
The /var/run directory is not preserved between reboots.
Then, when the VM reboots, the CNI binary may run before k8s creates the /var/run/azure-vnet directory. When the CNI binary runs first, it creates the directory with 0644 permissions. This causes permission denied errors for the cns. Even if k8s creates/mounts the /var/run/azure-vnet directory later, it will see it already exists and won't recreate the directory with the 0755 permissions.

What you expected to happen:
The CNI binary should create the directory with 0755 permissions.

How to reproduce it:
Reboot the VM with the cns capabilities security context dropping all capabilities (so it doesn't bypass permission checks). There is a chance that the azure-cns pod will get stuck in crash loop backoff.

Orchestrator and Version (e.g. Kubernetes, Docker):

Operating System (Linux/Windows):

Kernel (e.g. uanme -a for Linux or $(Get-ItemProperty -Path "C:\windows\system32\hal.dll").VersionInfo.FileVersion for Windows):

Anything else we need to know?:
[Miscellaneous information that will assist in solving the issue.]

[rbtr]

• @jpayne3506 can you update the CNS specs that we use in CI with

securityContext:
  capabilities:
    drop:
      - ALL
    add:
      - NET_ADMIN # only necessary for delegated IPAM/Cilium
      - NET_RAW # only necessary for delegated IPAM/Cilium

and make sure that we have a test that's rebooting the Node and verifying CNS functionality afterwards

[rbtr]

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods sounds like it would let the k8s chmod the directory when the CNS Pod mounts it as a Volume?

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity. Remove stale label or comment or this will be closed in 7 days

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity. Remove stale label or comment or this will be closed in 7 days

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity. Remove stale label or comment or this will be closed in 7 days

[QxBytes]

Node Rebooting Test added in: #2901

[jpayne3506]

Backport PR can be created for release/v1.5, but the validate package within release/v1.4 needs to be updated to consume the work. Will leave this open until both have been merged

• [backport] CI/CD changes to PR and CNI Release Test Pipelines #2940 release/v1.5

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity. Remove stale label or comment or this will be closed in 7 days

[rbtr]

@QxBytes looks like this was fully backported, is it good to resolve?

[jpayne3506]

#2901 did not get backported to release/v1.4 as the testing relies on the validation package being updated. Leaving this open until that work is complete and all E2E within this release train can validate Linux properly.

azure-container-networking/.pipelines/singletenancy/aks/e2e-step-template.yaml

Lines 62 to 75 in 73a0919

	- script: |
	kubectl get pods -A -o wide
	echo "Deploying test pods"
	cd test/integration/load
	go test -count 1 -timeout 30m -tags load -run ^TestLoad$ -tags=load -iterations=2 -scaleup=${{ parameters.scaleup }} -os=${{ parameters.os }}
	cd ../../..
	# Remove this once we have cniv1 support for validating the test cluster
	echo "Validate State skipped for linux cniv1 for now"
	if [ "${{parameters.os}}" == "windows" ]; then
	make test-validate-state OS=${{ parameters.os }}
	fi
	kubectl delete ns load-test
	displayName: "Validate State"
	retryCountOnTaskFailure: 3

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity. Remove stale label or comment or this will be closed in 7 days

[github-actions]

Issue closed due to inactivity.