Multi-threaded iperf3 public beta 2

[bmah888]

ESnet is pleased to announce the second PUBLIC BETA of a multi-threaded iperf3. This second beta fixes an error that caused excess CPU usage. It also contains a workaround that should allow compilation in build environments that do not support C11 atomic variables. (This includes CentOS 7 and related Linux distributions.)

To try this out, download the following release tarball (with SHA256 hash as shown):

https://downloads.es.net/pub/iperf/iperf-3.13-mt-beta2.tar.gz
83e482545b236acb7cef0bcc038129cc14c0326bd32a7475f0235a871625dc23 iperf-3.13-mt-beta2.tar.gz

Alternatively, clone the iperf3 repository as usual from GitHub https://github.com/esnet/iperf and switch to the mt branch.

Background

iperf3 was originally designed as a single-threaded program. Unfortunately, as network speeds increased faster than CPU clock rates, this design choice meant that iperf3 became incapable of using the bandwidth of the links in its intended operating environment (high-performance R&E networks with Nx10Gbps or Nx100Gbps network links and paths).

We have created a variant of iperf3 that uses a separate thread (pthread) for each test stream. As the streams run more-or-less independently, this should remove some of the performance bottlenecks, and allow iperf3 to perform higher-speed tests, particularly on 100+Gbps paths. This version has recorded transfers as high as 148Gbps in internal testing at ESnet.

The -P/--parallel option will vary the number of parallel streams as usual. However, multi-threaded iperf will allocate a separate thread to every stream/connection (on both client and server).

Most of the testing to date has taken place on Linux x86_64 bare-metal systems, although we've tested for basic operation on FreeBSD and macOS as well. Single-threaded and multi-threaded iperf3 should inter-operate seamlessly.

We intend, if all goes well, to merge this branch to the iperf3 mainline at some point in the future (still to be determined).

We're interested in hearing about people's experiences. Probably the best way is to add to the discussion at:

https://github.com/esnet/iperf/discussions/1496

We're especially interested in these areas for testing:

• All platforms: Best TCP and UDP performance on high-speed paths
  (100+Gbps).

• Virtual machines, particularly those with small numbers of virtual CPUs: We expect to see poor performance if the number of parallel streams exceeds the number of virtual CPUs, due to threads thrashing between CPU cores.

• 32-bit CPUs (for example i386 and 32-bit arm): Multi-threaded iperf3 now attempts to use atomic variables to hold various statistics and counters. On our test hosts, these are able to run without locking, so the various threads run more-or-less independently. We're not sure how well this translates to 32-bit hosts, which might need locking for these atomic variables.

• Non-Linux/UNIX hosts: Although we stand by our statement of support for Linux, macOS, and FreeBSD hosts only, we don't want to gratuitously break iperf3 on other platforms.

• Possible regression: We've observed some small (3-5%) slowdowns with single-stream TCP performance compared to the stock iperf3.