linghtning‐kit on VMWare

This wiki shows the VMWare VM settings for linghtning-kit.

We tested our kit on vSphere 8, ConnectX6, Nvidia RTX A6000 and followed Nvidia official page and VMWare article.

Guest OS

Our guest OS is ubuntu22.04 64bit with HWE kernel 6.5.

GPU

Set displayless

This is RTX A6000 specific setting. If you don't this, BAR1 size is 256MB and linghtning-kit cannot run. You need to check your BAR1 size for your GPU. Follow displaymodeselector instructions and turn the gpu to physical_display_disabled

CUDA Toolkit

We used 12.4 with open kernel.

VMWare VM settings

We set PCIPassthru settings below.

firmware=“efi"
pciPassthru.use64bitMMIO="TRUE"
pciPassthru.64bitMMIOSizeGB = “128"
pciPassthru.allowP2P = “TRUE”

64bitMMIOSizeGB depends on your GPU. see this table. allowP2P enables to access GPU BAR1 memory from NIC directly.