udp weird behavior with different speed and datagrams size

[chenr6000]

hi,
i'm having some weird behavior while using iperf with udp:
first, i'm changing both of my device to work with 10Mb/s speed.
i try to use iperf with datagram of 8Kb and that's my results:

iperf3 -c 1.0.0.1 -p 5201 -u -b 10mb -l 8kb

Accepted connection from 1.0.0.2, port 59962
[ 5] local 1.0.0.1 port 5201 connected to 1.0.0.2 port 46626
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 1.14 MBytes 9.57 Mbits/sec 0.497 ms 0/146 (0%)
[ 5] 1.00-2.00 sec 1.12 MBytes 9.37 Mbits/sec 0.557 ms 3/146 (2.1%)
[ 5] 2.00-3.00 sec 880 KBytes 7.21 Mbits/sec 0.664 ms 44/154 (29%)
[ 5] 3.00-4.00 sec 288 KBytes 2.36 Mbits/sec 0.489 ms 14/50 (28%)
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0.489 ms 0/0 (0%)
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0.489 ms 0/0 (0%)
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 0.489 ms 0/0 (0%)
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0.489 ms 0/0 (0%)
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0.489 ms 0/0 (0%)
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0.489 ms 0/0 (0%)

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.06 sec 3.40 MBytes 2.83 Mbits/sec 0.489 ms 61/496 (12%) receiver

later i try send datagram of 1472b (mtu size) and that's my results:

iperf3 -c 1.0.0.1 -p 5201 -u -b 10mb -l 1472

Accepted connection from 1.0.0.2, port 59970
[ 5] local 1.0.0.1 port 5201 connected to 1.0.0.2 port 59219
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 1.14 MBytes 9.56 Mbits/sec 0.320 ms 0/812 (0%)
[ 5] 1.00-2.00 sec 1.14 MBytes 9.56 Mbits/sec 0.311 ms 3/815 (0.37%)
[ 5] 2.00-3.00 sec 1.14 MBytes 9.57 Mbits/sec 0.357 ms 44/857 (5.1%)
[ 5] 3.00-4.00 sec 1.14 MBytes 9.57 Mbits/sec 0.458 ms 36/849 (4.2%)
[ 5] 4.00-5.00 sec 1.14 MBytes 9.57 Mbits/sec 0.346 ms 34/847 (4%)
[ 5] 5.00-6.00 sec 1.14 MBytes 9.56 Mbits/sec 0.362 ms 37/849 (4.4%)
[ 5] 6.00-7.00 sec 1.14 MBytes 9.57 Mbits/sec 0.376 ms 37/850 (4.4%)
[ 5] 7.00-8.00 sec 1.14 MBytes 9.57 Mbits/sec 0.364 ms 37/850 (4.4%)
[ 5] 8.00-9.00 sec 1.14 MBytes 9.56 Mbits/sec 0.353 ms 38/850 (4.5%)
[ 5] 9.00-10.00 sec 1.14 MBytes 9.57 Mbits/sec 0.353 ms 37/850 (4.4%)
[ 5] 10.00-10.06 sec 69.0 KBytes 9.36 Mbits/sec 0.344 ms 0/48 (0%)

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.06 sec 11.5 MBytes 9.57 Mbits/sec 0.344 ms 303/8477 (3.6%) receiver

later, i'm changing the speed of both device to 100Mb/s, and i get inverted results.
for 8Kb datagram:

iperf3 -c 1.0.0.1 -p 5201 -u -b 100mb -l 8kb

Accepted connection from 1.0.0.2, port 59976
[ 5] local 1.0.0.1 port 5201 connected to 1.0.0.2 port 51633
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 11.4 MBytes 95.5 Mbits/sec 0.689 ms 0/1458 (0%)
[ 5] 1.00-2.00 sec 11.4 MBytes 95.6 Mbits/sec 0.855 ms 0/1461 (0%)
[ 5] 2.00-3.00 sec 11.4 MBytes 96.2 Mbits/sec 0.968 ms 0/1465 (0%)
[ 5] 3.00-4.00 sec 11.4 MBytes 95.8 Mbits/sec 0.815 ms 0/1462 (0%)
[ 5] 4.00-5.00 sec 11.4 MBytes 95.6 Mbits/sec 1.106 ms 0/1463 (0%)
[ 5] 5.00-6.00 sec 11.4 MBytes 95.9 Mbits/sec 0.889 ms 0/1461 (0%)
[ 5] 6.00-7.00 sec 11.4 MBytes 95.7 Mbits/sec 1.025 ms 0/1461 (0%)
[ 5] 7.00-8.00 sec 11.4 MBytes 96.0 Mbits/sec 0.763 ms 0/1464 (0%)
[ 5] 8.00-9.00 sec 11.4 MBytes 95.7 Mbits/sec 0.896 ms 0/1462 (0%)
[ 5] 9.00-10.00 sec 11.4 MBytes 95.8 Mbits/sec 0.748 ms 0/1462 (0%)
[ 5] 10.00-10.01 sec 56.0 KBytes 55.7 Mbits/sec 0.829 ms 0/7 (0%)

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.01 sec 114 MBytes 95.8 Mbits/sec 0.829 ms 0/14626 (0%) receiver

but for 1472b datagram:

iperf3 -c 1.0.0.1 -p 5201 -u -b 100mb -l 1472

Accepted connection from 1.0.0.2, port 59988
[ 5] local 1.0.0.1 port 5201 connected to 1.0.0.2 port 53160
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 756 KBytes 6.17 Mbits/sec 3.880 ms 6647/7173 (93%)
[ 5] 1.00-2.00 sec 621 KBytes 5.11 Mbits/sec 0.295 ms 8377/8809 (95%)
[ 5] 2.00-3.00 sec 653 KBytes 5.33 Mbits/sec 7.827 ms 5819/6273 (93%)
[ 5] 3.00-4.00 sec 842 KBytes 6.90 Mbits/sec 5.325 ms 7309/7895 (93%)
[ 5] 4.00-5.00 sec 697 KBytes 5.71 Mbits/sec 8.574 ms 7871/8356 (94%)
[ 5] 5.00-6.00 sec 847 KBytes 6.94 Mbits/sec 4.398 ms 7294/7883 (93%)
[ 5] 6.00-7.00 sec 809 KBytes 6.63 Mbits/sec 3.669 ms 8362/8925 (94%)
[ 5] 7.00-8.00 sec 703 KBytes 5.75 Mbits/sec 0.674 ms 8841/9330 (95%)
[ 5] 8.00-9.00 sec 410 KBytes 3.36 Mbits/sec 8.444 ms 6069/6354 (96%)
[ 5] 9.00-10.00 sec 755 KBytes 6.18 Mbits/sec 8.019 ms 7631/8156 (94%)

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.01 sec 6.93 MBytes 5.80 Mbits/sec 8.019 ms 74220/79154 (94%) receiver

I've tried changing the datagram to 1Kb and saw a new issue:

iperf3 -c 1.0.0.1 -p 5201 -u -b 100mb -l 1kb

[ 5] local 1.0.0.1 port 5201 connected to 1.0.0.2 port 59769
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.01 sec 4.99 MBytes 41.6 Mbits/sec 0.529 ms 5664/10774 (53%)
[ 5] 1.01-2.00 sec 4.85 MBytes 40.9 Mbits/sec 0.271 ms 6502/11467 (57%)
[ 5] 2.00-3.00 sec 4.83 MBytes 40.4 Mbits/sec 0.564 ms 6563/11508 (57%)
[ 5] 3.00-4.00 sec 4.86 MBytes 40.8 Mbits/sec 0.518 ms 6478/11451 (57%)
[ 5] 4.00-5.00 sec 4.83 MBytes 40.4 Mbits/sec 0.571 ms 6521/11462 (57%)
[ 5] 5.00-6.01 sec 4.89 MBytes 40.8 Mbits/sec 0.565 ms 6520/11525 (57%)
[ 5] 6.01-7.01 sec 4.98 MBytes 41.8 Mbits/sec 0.552 ms 6363/11466 (55%)
[ 5] 7.01-8.01 sec 4.85 MBytes 40.7 Mbits/sec 0.548 ms 6453/11417 (57%)
[ 5] 8.01-9.00 sec 4.76 MBytes 40.1 Mbits/sec 0.231 ms 6636/11507 (58%)
[ 5] 9.00-10.00 sec 4.86 MBytes 40.8 Mbits/sec 0.250 ms 6422/11397 (56%)
[ 5] 10.00-10.05 sec 255 KBytes 40.4 Mbits/sec 0.438 ms 280/535 (52%)

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.05 sec 48.9 MBytes 40.8 Mbits/sec 0.438 ms 64402/114509 (56%) receiver

the results are better than the 1472 datagrams, but i got an error that my rx ring got full in the process (meaning that my controller wrote to the dram faster than my cpu was able to read).

i have 2 questions:

1. does anyone has any idea for the different behavior when i'm changing the speeds? (i was expecting that in different speeds i would get same behavior for same datagram size)
2. why my rx ring getting full faster with 1kb datagram than in 8kb/1472b datagram? (I haven't got this error for any of them).

thanks.

[bmah888]

It's a little tricky to speculate on what's happening here, we could use some more information like what operating system / distribution you're using and what version of iperf3 you have.

One possibility is that you're overrunning the socket buffers on either the sender (client) or receive (server) side. Perhaps try experimenting with the -w command line option to see if larger values (say 64K or larger) might reduce the loss?