❗ I like telling UDP jokes because I do care if you don't get them.
One of the main issues when using data diodes is not being able to check if the UDP packets you send arrived at the receiving proxy. During testing using large files we found out that one of the main issues is that the receiving application is not able to read the UDP packets quick enough from the rx_queue. In this overview we try to explain this.
Please add an issue for this article if you can help because we still haven't found a final solution for this problem.
In this overview we use 5 steps to explain the issue:
- Sender application sends/receive the data
- The application sends the data to the TX_QUEUE in kernel space
- The UDP packets are send over the cable trough the data diode to the receiver
- The receiver receives the UDP packets and places them in the RX_QUEUE
- The receiver application reads the UDP packets from the RX_QUEUE, processes them and writes the packets to the file
We found out that most of the packet loss happens on the receiving machine when the application is not able to read the RX_QUEUE fast enough. When the RX_QUEUE is full the kernel drops the packets and data is lost.
- Send the data slower from the sender. The applications UDPcast or PV (pipe viewer) are able to send the data on a lower speed.
- Enlarge the RX_QUEUE using sysctl. This only helps for small bursts, when sending multiple Gb's this has no use.
- Use FEC (Forward Error Control). The application UDPcast has the option to add FEC.
- Faster processing at the receiving application. Try to reduce the CPU load of the receiving application and/or limit other applications using the CPU
- Faster writing to disk. Use SSD or memdisk to write the data.
Monitor UDP queues and packets dropped. We are still looking into the best way to monitor the queues and this can be done on several ways.
Monitor both the sender and receiver.
cat /proc/net/udp
or use netstat
sender
sudo netstat -c -udp -an | grep "9001"
recording
receiver
sudo netstat -c -udp -an | grep ".255:9000"
recording
or use ss
watch -n 1 "ss -u -a -p -t '( sport = :9000 )'"
or use traffic control (tc).
tc -s qdisc show dev eth0
Bottom line, we noticed that the Recv-Q fills and the UDP-receiver application stops/crashed. Recv-Q is the count of bytes not copied by the user program connected to this socket.
Except for increasing the sysctl net.core.rmem_max we are still looking for a solution how to prevent the Recv-Q dropping packets. Please comment if you can help.
sudo sysctl -w net.core.rmem_max=32777216
Monitoring the softnet_stat we saw no increase of queues.
cat /proc/net/softnet_stat
https://www.cyberciti.biz/faq/linux-show-dropped-packets-per-interface-command/
sudo apt-get install libpcap-dev libnl-3-dev libnl-genl-3-dev binutils-dev libreadline6-dev autoconf libtool pkg-config build-essential```
git clone https://github.com/nhorman/dropwatch
cd dropwatch
./autogen.sh
./configure
make
sudo make install
check the current settings
cat /proc/sys/kernel/kptr_restrict
sudo su
echo 0 > /proc/sys/kernel/kptr_restrict
dropwatch -l list
sudo dropwatch -l kas
start
CPU - NIC Isolation: #10 First step in exploration..
Good story about tuning TCP. Perhaps lowering the garbage collection could help to reduce packet loss on the receiving side.
https://blog.cloudflare.com/the-story-of-one-latency-spike/
https://unix.stackexchange.com/questions/611981/udp-packet-drops
Helpful links
https://arthurchiao.github.io/blog/monitoring-network-stack/
https://www.sobyte.net/post/2022-05/linux-udp-packet-drop-debug/
https://iopscience.iop.org/article/10.1088/1748-0221/15/09/T09005/pdf
B System configuration
The following commands were used (performed as superuser) to change the system parameters on CentOS. The examples below modifies network interface eno49. This should be changed to match the name of the interface on the actual system.
sysctl -w net.core.rmem_max=12582912
sysctl -w net.core.wmem_max=12582912
sysctl -w net.core.netdev_max_backlog=5000
ifconfig eno49 mtu 9000 txqueuelen 10000 up
More details about UDP tuning:
https://gilbertasm.github.io/2018/09/13/tunning-udp-buffers.html
https://blog.packagecloud.io/monitoring-tuning-linux-networking-stack-sending-data/#monitoring-udp-protocol-layer-statistics