The support for GSO/GRO features

[nathan565656]

The Linux kernel has supported GSO (Generic Segmentation Offload) and GRO (Generic Receive Offload) features, and using these features reduces the processing overhead on the UDP applications when supporting high data rates. The latest Android framework also provides applications the options to use the GSO and GRO option to move data with less processing overhead between socket and Kernel.

The attached code (based on version 3.7) shows how the existing iperf3 could be extended to support GRO/GSO options. The sample code was verified on Ubuntu 20.04 (where kernel version is 5.4.0-90-generic and GSO/GRO is supported). In the experiment, the application was verified for what data rates (100 Mbps and 1 Gbps rates) and could see the reduced iperf3 CPU usage.

It would be great if we can considered making this option available in the iperf3 so the users can take advantage of this feature.
iperf37_GSO-GRO-support.zip

[davidBar-On]

Hi @nathan565656, seems interesting. Can you please clarify few points about this approach

1. Isn't GSO/GRO implemented by the operating system or the UDP stack, so the application should not have to handle it?
2. If I understand your code properly, the support of GSO/GRO is hardcoded. Is there a way to dynamically detect whether these features are supported, the related buffer size, etc.?
3. What are the main differences between supporting GSO/GRO as in your code and just increasing the buffer size using the -l option?

Thanks

[nathan565656]

Hi @davidBar-On , thanks for the feedback. Please find the comment below.

1.) Yes. As you mentioned, GSO/GRO is implemented by the operating system network stack. However, it requires the application to configure socket GSO/GRO flags accordingly.

Tx path:

1. iperf_udp_gso() -- Enable GSO support by configurating socket GSO flags with the datagram size (as the function parameter) to tell the network stack the datagram size value used for fragmentation.
2. iperf_udp_send() -- Use the big-size buffer (ex. 1460 * 43 = 62780). This will allow application to write large amount of data in single write call. This reduce MIPS/cycles required by iperf3 to transmit data.

Rx path:

1. iperf_udp_gro() -- Enable GRO support by configurating socket GRO flags.
2. iperf_udp_recv() -- Use the big-size buffer (ex. 1460 * 43 = 62780) for data read to reduce the count of reading operations and hence MIPS/cycles are reduced.

2.) The following two points will check the operation system network stack capability.

1. “iperf_udp_connect()” -- Use “iperf_udp_gso()” to detect GSO capability by the socket flag “UDP_SEGMENT”.
2. “iperf_udp_accept()” -- Use “iperf_udp_gro()” to detect GRO capability by the socket flag “UDP_GRO”.

If GSO or GRO is not supported by the network stack, the socket flag configuration will be failed, and iperf3 will behaves as the original case.
If the intention is to control GSO/GRO support in iperf3, maybe we can have separate options (ex. --gso <0 or 1>, --gro <0 or 1>) to enable/disable the support.

3.) Simply increasing “-l” value will cause IP fragmentation. To use GSO/GRO feature in the network stack, requires the application:

1. Configure socket GSO/GRO flags (and tell the network stack the datagram size for GSO).
2. Use big-size buffer for data read/write to reduce the number of read/write operations to get full GSO/GRO benefits.
3. It can get the best efficiency when having the buffer size used for data read/write operations same as the multiple of datagram size.

Please feel free to correct if any misunderstanding, and please feel free to let me know if any other questions.

[davidBar-On]

Hi @nathan565656, thanks for the detailed explanation. I was not aware that GSO/GRO is implemented by the UDP stack and not by the NIC as TSO.

Are you planning to send a Push Request with these enhancements to add GRO/GSO support to iperf3?

In any case, following some comments to the enhancements:

1. QC_GRO / QC_GSO support should be determined dynamically at compile time, according to whether UDP_GRO / UDP_SEGMENT are defined. This should be done in configure.ac. Note that configure (and maybe bootstrap.sh) should re-run when configure.ac is changed. ( Just disabling the features when setsockopt with these options fail is o.k.)
2. An iperf3 option should be added to enable GRO and GSO (settings->gro/gso), with the default that they are not used. To minimize number of additional options for iperf3, it may be sufficient to enable both GSO and GRO using one option --gsro (will set both ). If separation between enabling each will be needed in the future, it will be possible to add a value (e.g. --GSRO R to enable only GRO).
3. If I understand correctly lines lines 36-46, it is assumed that the client is the sender and the server is the receiver. With the reverse (-R) mode, the server can be the sender, so both GRO and GSO options should be passed to the server.

[nathan565656]

Hi @davidBar-On , thanks a lot for your suggestion and guidance!

The Push Request 1309 has been submitted with the following enhancement.

• Add the support in configure.ac to detect if the platform can support GSO/GRO (so the related codes will be included/removed during the compiling time).
• Add a new option “--gsro" to enable GSO/GRO.
• Add the support for the reverse (-R) mode.
• Some changes for datagram statistics.

Please feel free to correct or let me know if anything is missing or any enhancement.
Thanks.

[nathan565656]

Hi @davidBar-On, if you still see anything is missing in the push request or it's better to modify something to make it better, please feel free to let me know. I will try to update soon.

Hope we can have this support in the iperf3 in the near future if possible. Because GSO/GRO support is more and more common on many platforms now, especially for the embedded-device products where MIPS is a big concern. With this support, iperf3 users can take this advantage to profile the real MIPS counts and thermal status as well, and maybe get better peak throughput result if the bottleneck is at the platform performance.

Appreciate your suggestion and guidance always!

[davidBar-On]

Hi @nathan565656, I don't see anything else that is missing. However, I am not part of the team that maintain iperf3 and decides which PRs to include, so I cannot help that.

[nathan565656]

Hi @davidBar-On, thanks for your review!

[ygblue]

GSO and GRO benefits Windows too. With this change iperf3 compiled with Cygwin can get much higher throughput and lower CPU utilization compared to the default iperf3 which has no GSO and GRO on Windows. This feature is especially useful to test the NIC which supports 5G bps or 10G bps or even above while default iperf3 with Cygwin is hard to reach such throughput on UDP.

[gegles]

@nathan565656 or @marcosfsch, any progress on getting this work updated and merged?

[gbhardwaja]

Pl help share when this feature will be merged?