Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ontap-san with nvme breaks when /proc/mounts is excessive #896

Open
magicite opened this issue Mar 28, 2024 · 0 comments
Open

ontap-san with nvme breaks when /proc/mounts is excessive #896

magicite opened this issue Mar 28, 2024 · 0 comments

Comments

@magicite
Copy link

Describe the bug
Pods requesting NVMeoF-backed PVCs fail to have the storage attached to their nodes on nodes where the /proc/mounts content is lengthy. When this occurs, the nvme command emits something to stderr. I believe Trident is erroneously capturing the stderr message and failing.

cn1001:~ # nvme list
libhugetlbfs: ERROR: Line too long when parsing mounts
Node                  Generic               SN                   Model                                    Namespace Usage                      Format           FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            81MrpNW3EewqAAAAAAAB NetApp ONTAP Controller                  1          53.69  GB /  53.69  GB      4 KiB +  0 B   FFFFFFFF

On the node itself, I can manually (i.e., without trident) attach to the storage over NVMeoF, format the block device, etc.

Environment
Provide accurate information about the environment to help us reproduce the issue.

  • Trident version: 24.02.0
  • Container runtime: containerd://1.7.11-k3s2
  • Kubernetes version: v1.27.10+rke2r1
  • Kubernetes orchestrator: Harvester v1.3.0
  • OS: SLE micro 5.4
  • NetApp backend types: ONTAP A150

To Reproduce

  1. Prepare a k8s cluster with connectivity to a NetApp with NVMeoF support
  2. Configure the cluster with trident and configure a backend with ontap-san sanType nvme
  3. Create a mount entry such that it shows up in /proc/mounts with a line size greater than 2048, which is enough to cause the libhugetlbfs stderr (reference)
  4. Create a storage class that will target the previously created backend
  5. Create a PVC referencing the storage class
  6. Create a pod referencing the PVC

Expected behavior
The PVC should be dynamically provisioned, the pod should be scheduled to a node, the storage should attach to the node, and the pod should run with access to the storage.

Additional context
Logs attached as gathered from using tridentctl logs -n trident --node cn1003 --archive --sidecars
support-2024-03-28T10-31-01-CDT.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants