The repository contains the rdma-unit-test framework. rdma-unit-test is an open-source collection of unit tests to extensively test an ibverbs library/driver.
These are single-machine single-nic tests which focus on stressing the ibverbs interface. The NIC must be able to issue loopback operations to itself (via internal loopback, bouncing off a switch, or any other setup which reflects the packets). The tests assume global routing. See flags for ways to control IP version and MTU.
Explicitly out of scope is stressing the fabric and transport.
These are practical tests designed to understand the behavior of a stack. In many places they check constraints beyond what is stated in the Infiniband specification. Do not assume test failures indicate bad hardware, it could be that your device diverged in an undefined area of the specification. Patches to fix overly-specified tests are encouraged.
rdma-unit-test has been built/tested on CentOS 8.3.2011 using a Mellonox ConnectX-4 Dual port 25Gbe adapter. Installation requirements may vary depending upon the distro used. CentOS installation uses the standard Server configuration with the following options;
- Infiniband support
- Development Tools
Additional requirements;
- ivberbs development libraries; sudo yum install libibverbs-devel
- bazel build tool; instructions here
The user space libraries are supported and packaged here
Introspection is used to selectively enable tests depending on hardware capabilities. device_attr is incomplete and miss-reported by some NICs. In many cases introspection dictates expected behavior in undefined areas of the specification. Adding introspection support for a new NIC requires 2 changes:
- Extend NicIntrospection (ex. introspection_rxe.h)
- Update gunit_main.cc to register the new introspection.
First clone rdma-unit-test into a workspace directory.
git clone https://github.com/google/rdma-unit-test.git
Change to the project directory and run the tests. The following will recursively build and execute all the test targets.
cd rdma-unit-test
bazel test ... [--test_output=all] [--test_arg=<flag>]
An example using SoftRoce;
cd rdma-unit-test
bazel test ... --test_output=all --test_arg=--verbs_mtu=1024 --test_arg=--no_ipv6_for_gid
It has been observed that some ibverb libraries require root priviledges to create verb objects. If this is the case you can either run bazel under root or execute each individual test.
bazel's caches the workspace so if you run bazel as root it will rebuild the targets as root. In the project directory;
cd rdma-unit-test
sudo bazel test ... [--test_output=all] [--test_arg=<flag>]
The tests are divided into categories in the 'cases' directory. To run a specific category of tests run bazel test in the 'cases' directory;
cd rdma-unit-test/cases
bazel test device_test [--test_output=all] [--test_arg=<flag>]
You can also provide provide a filter to only run specific test within a category. In the cases directory;
cd rdma-unit-test/cases
bazel test device_test [--test_output=all] [--test_arg=<flag>]
You can also run the test executables directly. This method does not support bazel test flag options; e.g. --test_output, --test_arg
cd rdma-unit-test/bazel-bin/cases
./device_test [--verbs_mtu=1024] [--no_ipv6_for_gid]
Several command line flags select specific behaviour to accomodate adapter limitations or behaviour modification;
| flag | default | description |
|---|---|---|
| verbs_mtu | 4096 | Changes the mtu size for qp configuration. Note that some adapters may have limited support for 4096 so this flag must be set to be within device contraints |
| no_ipv6_for_mtu | false | If set, will only enumerate ports with ipv4 addresses. |
| device_name | none | If set, will attempt to open the named device for all tests. If the device is not found rdma_unit_test will list the available devices. |
rdma-unit-test uses ibv_get_device_list to find all available devices. By default rdma-unit-test will use the first device found with an active port. To select a difference device name or port use the --device_name flag.
This package has been tested on the following adapters;
- Mellonox ConnectX-3
- Mellonox ConnectX-4 (may require --verbs_mtu=1024)
- SoftRoce (limited support, requires --no_ipvp6_for_gid --verbs_mtu=1024)