Add support for SKIP-RX-COPY using MSG_TRUNC and Zero-copy using SO_ZEROCOPY/MSG_ZEROCOPY #1690

davidBar-On · 2024-04-27T15:06:30Z

Version of iperf3 (or development branch, such as master or
3.1-STABLE) to which this pull request applies:
master
Issues fixed (if any): feature request: support for MSG_ZEROCOPY and MSG_TRUNC flags #1678
Brief description of code changes (suitable for use as a commit message):

Add support for SKIP-RX-COPY (using MSG_TRUNC) and SO_ZEROCOPY/MSG_ZEROCOPY. Although it is not clear that all added functionality improve performance and throughput, support for all these options and their combinations was added, to allow testing all of them. The assumptions is that different environments may have different levels of support for the different options and their combinations.

(Note that running bootstrap.sh; configure is required before make to support the new features.)

The added options are:

--skip-rx-copy: when used, for both TCP and UDP, recv(..., MSG_TRUNC) is used instead of read().
Support for MSG_ZEROCOPY. When used, socket option SO_ZEROCOPY is set and send(...., MSG_ZEROCOPY) is used instead of write(). MSG_ZEROCOPY is used in the following cases:
2.1 UDP: when -Z/--zerocopy option is set.
2.2 TCP: when --zerocopy=z is set. Otherwise, sendfile() continue to be used for TCP zero copy.

marcosfsch · 2024-04-30T00:20:59Z

Thanks David, this looks very promissing.

Follows my results on a 100G back to back test environment:

Server command: sudo taskset --cpu-list 1 ./src/iperf3 -s -i 1
Client Comands:
sudo taskset --cpu-list 1 ./src/iperf3 -c 192.168.1.18 -i 1 -t 30 -V
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 126 GBytes 36.0 Gbits/sec 8 sender
[ 5] 0.00-30.00 sec 126 GBytes 36.0 Gbits/sec receiver
CPU Utilization: local/sender 40.2% (0.9%u/39.3%s), remote/receiver 99.7% (1.7%u/98.0%s)

sudo taskset --cpu-list 1 ./src/iperf3 -c 192.168.1.18 -i 1 -t 30 -V --skip-rx-copy
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 239 GBytes 68.4 Gbits/sec 0 sender
[ 5] 0.00-30.00 sec 239 GBytes 68.4 Gbits/sec receiver
CPU Utilization: local/sender 79.5% (1.8%u/77.8%s), remote/receiver 90.5% (4.4%u/86.0%s)

sudo taskset --cpu-list 1 ./src/iperf3 -c 192.168.1.18 -i 1 -t 30 -V --skip-rx-copy --zerocopy=z
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 344 GBytes 98.5 Gbits/sec 0 sender
[ 5] 0.00-30.00 sec 344 GBytes 98.5 Gbits/sec receiver
CPU Utilization: local/sender 33.2% (2.1%u/31.1%s), remote/receiver 99.9% (6.4%u/93.5%s)

sudo taskset --cpu-list 1 ./src/iperf3 -c 192.168.1.18 -i 1 -t 30 -V --skip-rx-copy -Z
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 346 GBytes 99.0 Gbits/sec 0 sender
[ 5] 0.00-30.00 sec 346 GBytes 99.0 Gbits/sec receiver
CPU Utilization: local/sender 52.5% (2.2%u/50.3%s), remote/receiver 100.0% (6.5%u/93.4%s)

bltierney · 2024-05-03T13:59:16Z

This is fantastic! Thanks! I get 2x throughput on a 100G path.

Question: Shouldn't --skip-rx-copy be a server side option instead of a client side? (or both if using the -R option)

davidBar-On · 2024-05-03T14:51:26Z

Question: Shouldn't --skip-rx-copy be a server side option instead of a client side? (or both if using the -R option)

The client send this option (as several of the other options) to the server, so setting it for a test is applicable for both normal and reverse (-R) modes. Making it a server option means that that there will be no way to set the option per test when the server is a receiver.

bltierney · 2024-05-03T17:10:08Z

Ah, that makes sense. Thanks. I do see a use case where I might want to force it on the server side, but passing the option from the client is more useful.

davidBar-On · 2024-05-03T17:27:43Z

I now understand why you asked to have the option on the server side, a use case that I didn't think about. I am not sure it is better to try adding such enhancement to this PR, or wait first that this (or other similar) PR will me merged and then suggest the enhancement by additional PR. UPDATE: for now, I will not add the `skip-rx-copy` to the server options, unless the reviewers will recommend differently. This is because such setting will have different behavior for the server as a receiver and for the client as a receiver.

…

On Fri, May 3, 2024 at 8:10 PM Brian Tierney ***@***.***> wrote: Ah, that makes sense. Thanks. I do see a use case where I might want to force it on the server side, but passing the option from the client is more useful. — Reply to this email directly, view it on GitHub <#1690 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOSCPP2NT3L7K4JHPEKHOU3ZAPAILAVCNFSM6AAAAABG4FECFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJTGQZTEOBYGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

bmah888 · 2024-05-20T18:50:26Z

Thanks for the pull request! We're gonna need to study this a bit.

MattCatz · 2024-06-10T21:51:20Z

Having read through the patch (haven't run anything yet) and associated Linux documentation:

The MSG_TRUNC seems to affect TCP and UDP different. You have things set-up in a way that I would expect only TCP throughput to be improved. Has anyone seen an improvement in UDP throughput?
This iteration of changes is probably incompatible with the --file flag (specifically MSG_ZEROCOPY). To be compatible we would need to wait for the kernel to notify us that it is done with the shared buffer.

https://man7.org/linux/man-pages/man2/recv.2.html
https://man7.org/linux/man-pages/man7/tcp.7.html
https://docs.kernel.org/networking/msg_zerocopy.html

MattCatz · 2024-06-10T22:06:56Z

src/iperf_api.c

+    // sero copy for TCP use sendfile()
+    if (test->zerocopy && test->protocol->id != Pudp && !has_sendfile()) {
+        i_errno = IENOSENDFILE;
+        return -1;
+    }
+#else
+    // sero copy is supported only by TCP


Suggested change

// sero copy for TCP use sendfile()

if (test->zerocopy && test->protocol->id != Pudp && !has_sendfile()) {

i_errno = IENOSENDFILE;

return -1;

}

#else

// sero copy is supported only by TCP

// zero copy for TCP use sendfile()

if (test->zerocopy && test->protocol->id != Pudp && !has_sendfile()) {

i_errno = IENOSENDFILE;

return -1;

}

#else

// zero copy is supported only by TCP

Changed (with other changes - see comments)

davidBar-On · 2024-06-11T19:24:35Z

The MSG_TRUNC seems to affect TCP and UDP different. You have things set-up in a way that I would expect only TCP throughput to be improved. Has anyone seen an improvement in UDP throughput?

Why "things set-up in a way that I would expect only TCP throughput to be improved"? Although practically that may be the case, what is wrong in the implementation regarding UDP? (Both UDP and TCP use Nrecv() with MSG_TRUNC as socket-option.)

This iteration of changes is probably incompatible with the --file flag (specifically MSG_ZEROCOPY). To be compatible we would need to wait for the kernel to notify us that it is done with the shared buffer.

Forgot to take --file into account. Without this option, the sent buffer is fixed, so there is no need to handle the kernel notifications. To simplify the initial solution, I now added a check to not allow the use of MSG_ZEROCOPY when --file is set.

MattCatz · 2024-06-11T20:04:50Z

Why "things set-up in a way that I would expect only TCP throughput to be improved"? Although practically that may be the case, what is wrong in the implementation regarding UDP? (Both UDP and TCP use Nrecv() with MSG_TRUNC as socket-option.)

If I understand the Linux kernel documentation correctly, recv(fd, buf, nleft, MSG_TRUNC) will discard different parts of the kernel's buffer depending on if it is UDP or TCP.

With TCP it will discard nleft from the kernel buffer.

With UDP it will discard everything after nleft.

This means that we only have to read the first part of UDP packet to get the UDP stats the server sticks in there:

--- a/src/iperf_udp.c
+++ b/src/iperf_udp.c
@@ -69,6 +69,9 @@ iperf_udp_recv(struct iperf_stream *sp)
     sock_opt = 0;
 #endif /* HAVE_MSG_TRUNC */
 
+    if (sock_opt)
+        size = sizeof(sec) + sizeof(usec) + sizeof(pcount);
+
     r = Nrecv(sp->socket, sp->buffer, size, Pudp, sock_opt);
 
     /*

--- a/src/net.c
+++ b/src/net.c
@@ -397,8 +397,8 @@ Nread(int fd, char *buf, size_t count, int prot)
 int
 Nrecv(int fd, char *buf, size_t count, int prot, int sock_opt)
 {
-    register ssize_t r;
-    register size_t nleft = count;
+    register ssize_t r, total=0;
+    register ssize_t nleft = count;
     struct iperf_time ftimeout = { 0, 0 };
 
     fd_set rfdset;
@@ -441,6 +441,7 @@ Nrecv(int fd, char *buf, size_t count, int prot, int sock_opt)
         } else if (r == 0)
             break;
 
+        total += r;
         nleft -= r;
         buf += r;
 
@@ -477,7 +478,7 @@ Nrecv(int fd, char *buf, size_t count, int prot, int sock_opt)
             }
         }
     }
-    return count - nleft;
+    return total;
 }

Doing a quick loopback test on my system shows about a 20% increase in throughput for large UDP packets.

On a quick tangent, it looks like with this change UDP tests are pretty much dominated by the overhead of select.

Below is not using --skip-rx-copy:

Below is using --skip-rx-copy:

MattCatz · 2024-06-11T20:11:25Z

Forgot to take --file into account. Without this option, the sent buffer is fixed, so there is no need to handle the kernel notifications. To simplify the initial solution, I now added a check to not allow the use of MSG_ZEROCOPY when --file is set.

I wasn't aware of this in my original comment but since iperf likes to insert UDP stats at the start of the buffer used for UDP packets, using MSG_ZEROCOPY will create a similar race condition for UDP tests. This will cause iperf to miss report lost datagrams.

davidBar-On · 2024-06-14T13:52:37Z

If I understand the Linux kernel documentation correctly, recv(fd, buf, nleft, MSG_TRUNC) will discard different parts of the kernel's buffer depending on if it is UDP or TCP. ... With UDP it will discard everything after nleft. This means that we only have to read the first part of UDP packet to get the UDP stats the server sticks in there:

Thanks a lot! I completely missed that point. I now added you suggested change to iperf_udp_recv(). I didn't make the suggested changes to Nrecv(), since I understand that they are just cosmetic, and usually my approach is to make the minimum changes required, as this seems to be more safe (bugs introduction, portability, etc.).

The new commit is with rebase, and I forgot to mention previously that the changes include the PR #1708 fix, to reduce server's CPU overhead.

... UDP stats at the start of the buffer used for UDP packets, using MSG_ZEROCOPY will create a similar race condition for UDP tests.

As you probably understood, I somehow overlooked the UDP dynamic prefix of a packet ... For the receiving side you solved the issue with the above. For the sending side it seems that the only solution is using the MSG_ZEROCOPY Notifications, but I don't want to add this complexity at this point. It seems that initially it is better to just not implement zero copy for UDP. Before doing that, do you agree? Do you have other suggestion?

MattCatz · 2024-06-14T15:25:38Z

I agree that MSG_ZEROCOPY won't work with UDP without notifications.

The math for Nread/Nrecv took me a second to reason through (because of the double negative). I.e With UDP_MAX reads you get something like:

count = 16;
nleft = count;
...
nleft -= 65507; // nleft = (-65491)
...
return count - nleft;  // 16 - (-65491) = 65507

(which does account for all the bits lol)

davidBar-On · 2024-06-16T18:29:26Z

The changes to support SKIP-RX-COPY where moved to PR #1717, and this PR will be used only for the support of zero-copy using SO_ZEROCOPY/MSG_ZEROCOPY.

The math for Nread/Nrecv took me a second to reason through (because of the double negative). ....

I am not sure I realized that nleft becomes negative, so for clarity I did make the suggested changes in PR #1717.

I agree that MSG_ZEROCOPY won't work with UDP without notifications.

Will add support for the notifications in this PR.

davidBar-On · 2024-06-23T18:52:45Z

Added a separate PR #1720 for the support of MSG_ZEROCPY/SOZEROCOPY, now with notifications support.

Closing this PR as its functionality is now split between PRs #1717 and #1720.

swlars · 2024-10-25T18:19:58Z

@MattCatz Hi MattCatz, what were you using for the "flame graphs" for performance? Thanks!

MattCatz · 2024-10-28T16:27:01Z

@MattCatz Hi MattCatz, what were you using for the "flame graphs" for performance? Thanks!

I didn't write down the exact steps but probably something like:

build iperf3 with debug symbols and frame pointer (i.e. -fno-omit-frame-pointer)
install kernel debug symbols ( I believe on ubuntu it is something like sudo apt-get install linux-image-uname -r-dbgsym)
Follow this guide: https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
Run your run iperf3. e.x. perf -F 97 -- iperf3 -c 127.0.0.1

If that doesn't work then feel free to reach out over email (you can find it in the git log).

swlars · 2024-11-07T19:16:15Z

@MattCatz Thanks for the instructions! I'll give it a shot!

davidBar-On mentioned this pull request Apr 27, 2024

feature request: support for MSG_ZEROCOPY and MSG_TRUNC flags #1678

Open

MattCatz reviewed Jun 10, 2024

View reviewed changes

Changes per reviewer comments receive UDP prefix

abf359a

davidBar-On force-pushed the issue-1678-MSG_TRUNC_and_SO_ZEROCOPY-MSG_ZEROCOPY branch from 95ee7db to abf359a Compare June 14, 2024 13:30

davidBar-On mentioned this pull request Jun 16, 2024

Add SKIP-RX-COPY support - using MSG_TRUNC socket option #1717

Open

davidBar-On changed the title ~~Add support for SKIP-RX-COPY and SO_ZEROCOPY/MSG_ZEROCOPY~~ Add support for SO_ZEROCOPY/MSG_ZEROCOPY Jun 16, 2024

davidBar-On changed the title ~~Add support for SO_ZEROCOPY/MSG_ZEROCOPY~~ Add support for Zero-copy using SO_ZEROCOPY/MSG_ZEROCOPY Jun 16, 2024

davidBar-On changed the title ~~Add support for Zero-copy using SO_ZEROCOPY/MSG_ZEROCOPY~~ Add support for SKIP-RX-COPY using MSG_TRUNC and Zero-copy using SO_ZEROCOPY/MSG_ZEROCOPY Jun 23, 2024

davidBar-On mentioned this pull request Jun 23, 2024

Add SO_ZEROCOPY and MSG_ZEROCOPY support for UDP and TCP #1720

Open

davidBar-On closed this Jun 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for SKIP-RX-COPY using MSG_TRUNC and Zero-copy using SO_ZEROCOPY/MSG_ZEROCOPY #1690

Add support for SKIP-RX-COPY using MSG_TRUNC and Zero-copy using SO_ZEROCOPY/MSG_ZEROCOPY #1690

davidBar-On commented Apr 27, 2024 •

edited

Loading

marcosfsch commented Apr 30, 2024

bltierney commented May 3, 2024

davidBar-On commented May 3, 2024

bltierney commented May 3, 2024

davidBar-On commented May 3, 2024 via email •

edited

Loading

bmah888 commented May 20, 2024

MattCatz commented Jun 10, 2024

MattCatz Jun 10, 2024

davidBar-On Jun 11, 2024

davidBar-On commented Jun 11, 2024

MattCatz commented Jun 11, 2024

MattCatz commented Jun 11, 2024

davidBar-On commented Jun 14, 2024

MattCatz commented Jun 14, 2024

davidBar-On commented Jun 16, 2024

davidBar-On commented Jun 23, 2024

swlars commented Oct 25, 2024

MattCatz commented Oct 28, 2024

swlars commented Nov 7, 2024

Add support for SKIP-RX-COPY using MSG_TRUNC and Zero-copy using SO_ZEROCOPY/MSG_ZEROCOPY #1690

Add support for SKIP-RX-COPY using MSG_TRUNC and Zero-copy using SO_ZEROCOPY/MSG_ZEROCOPY #1690

Conversation

davidBar-On commented Apr 27, 2024 • edited Loading

marcosfsch commented Apr 30, 2024

bltierney commented May 3, 2024

davidBar-On commented May 3, 2024

bltierney commented May 3, 2024

davidBar-On commented May 3, 2024 via email • edited Loading

bmah888 commented May 20, 2024

MattCatz commented Jun 10, 2024

MattCatz Jun 10, 2024

Choose a reason for hiding this comment

davidBar-On Jun 11, 2024

Choose a reason for hiding this comment

davidBar-On commented Jun 11, 2024

MattCatz commented Jun 11, 2024

MattCatz commented Jun 11, 2024

davidBar-On commented Jun 14, 2024

MattCatz commented Jun 14, 2024

davidBar-On commented Jun 16, 2024

davidBar-On commented Jun 23, 2024

swlars commented Oct 25, 2024

MattCatz commented Oct 28, 2024

swlars commented Nov 7, 2024

davidBar-On commented Apr 27, 2024 •

edited

Loading

davidBar-On commented May 3, 2024 via email •

edited

Loading