PVCs fail to mount on a node but it previously worked - context deadline exceeded

[rusLukasRath]

Describe the bug
Trident PVCs could be mounted as normal on the worker node, but after some time or because of some unknown reason, Trident PVCs stop being able to be mounted on this exact node. Pods that are trying to mount a Trident PVC get the error message: "context deadline exceeded"

The exact same PVC can still be mounted on other worker nodes. This issue happens with all Trident PVCs, old and newly created after the issue started. Restarting the trident-node pod on said worker node does not fix the issue.

Trying to mount the NetApp shares manually on said node works completly fine.

Environment

Provide accurate information about the environment to help us reproduce the issue.

• Trident version: 24.02
• Trident installation flags used: Trident Operator with default values
• Container runtime: CRI-O 1.28.8-2.rhaos4.15.gitfcaab07.el9
• Kubernetes version: v1.28.11
• Kubernetes orchestrator: OpenShift v4.15.23
• Kubernetes enabled feature gates:
• OS: Red Hat Core OS with RHEL 9.2
• NetApp backend types: ontap-nas & ontap-nas-economy
• Other:

To Reproduce

Unknown

Expected behavior

Trident PVCs should be able to be mounted at all times.

Additional context

The cluster on which this problem occures is running all of our GitLab Runner build jobs. On this cluster dozens of build jobs are running simultaneously and multiple build jobs are starting at the same time that want to mount the same Trident PVCs.

Attached is the log of the trident-node pod on the node before we terminated and started a new one.
trident-node-linux-t5f54.txt