Incorrect checksums for ipsec

[bartekl]

Hi,
I'm running some POC using loxilb and reached a blocker for checksums not being correct for some packets.
Loxilb setup using https://github.com/loxilb-io/loxilbdocs/blob/main/docs/standalone.md (but w/ ipvlan interface instead of macvlan and inside a virtio based VM, given limitations of my cloud provider), simple basic nat LB policy for ports 500 and 4500 udp - ipsec, tunnel is established, but then you can see this at the receiver side:

07:00:22.682902 fe:00:04:e2:c2:05 > 56:00:04:e2:c2:05, ethertype IPv4 (0x0800), length 178: (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), length 164)
    A.C.45.83.4500 > B.D.245.251.4500: [bad udp cksum 0xa624 -> 0xcb06!] UDP-encap: ESP(spi=0xc5848d6d,seq=0x8), length 136
        0x0000:  4500 00a4 0000 4000 3811 2594 aacc 2d53  E.....@.8.%...-S
        0x0010:  bbdd f5fb 1194 1194 0090 a624 c584 8d6d  ………..$…m
        0x0020:  0000 0008 97f5 293d 61dd 028d 8215 7c53  ......)=a.....|S
        0x0030:  d043 c827 3b4f 8232 f617 a0f7 0b7d e31b  .C.';O.2.....}..
        0x0040:  6c21 244b 5b0e 03d8 7558 3003 17a7 c8c8  l!$K[...uX0.....
        0x0050:  fa71 e7a0 4cf0 d188 1529 c6c2 d9b6 58c3  .q..L....)....X.
        0x0060:  727a 6fb0 4096 7dca 5e83 008b 0819 9c21  rzo.@.}.^......!
        0x0070:  3091 3f1d 3370 3d33 f88c ab5f 8450 26d5  0.?.3p=3..._.P&.
        0x0080:  ee3e 84f7 2133 ea68 3a0d 0ce0 2798 733f  .>..!3.h:...'.s?
        0x0090:  c8ee 57cd 2c58 ce7f 1452 c2ea c86e 1ed5  ..W.,X...R...n..
        0x00a0:  60ee ae6f                                `..o
07:00:22.964237 fe:00:04:e2:c2:05 > 56:00:04:e2:c2:05, ethertype IPv4 (0x0800), length 174: (tos 0x0, ttl 56, id 63812, offset 0, flags [DF], proto UDP (17), length 160)
    A.C.45.83.4500 > B.D.245.251.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 0000027a cookie 37f3d0a8acc39c15->7b87c0f7f6c3fe4f: child_sa  inf2[I]:
    (v2e: len=96 3d9e0a2b403c2fda83d8e08a3bb6bcecbc06ee027f2e4e19ea883f1b7ad92727d44d215ae9dbb54ef4577b4e0634c2bb10d5a6041b86dc58e6d3b9b4a4b3e163df359e480b14b10d48ceafbd8fed3b273923d2f89455af8045a895037111ed80)
        0x0000:  4500 00a0 f944 4000 3811 2c53 aacc 2d53  E....D@.8.,S..-S
        0x0010:  bbdd f5fb 1194 1194 008c 3b5a 0000 0000  ……….;Z….
        0x0020:  37f3 d0a8 acc3 9c15 7b87 c0f7 f6c3 fe4f  7.......{......O
        0x0030:  2e20 2508 0000 027a 0000 0080 2900 0064  ..%....z....)..d
        0x0040:  3d9e 0a2b 403c 2fda 83d8 e08a 3bb6 bcec  =..+@</.....;...
        0x0050:  bc06 ee02 7f2e 4e19 ea88 3f1b 7ad9 2727  ......N...?.z.''
        0x0060:  d44d 215a e9db b54e f457 7b4e 0634 c2bb  .M!Z...N.W{N.4..
        0x0070:  10d5 a604 1b86 dc58 e6d3 b9b4 a4b3 e163  .......X.......c
        0x0080:  df35 9e48 0b14 b10d 48ce afbd 8fed 3b27  .5.H....H.....;'
        0x0090:  3923 d2f8 9455 af80 45a8 9503 7111 ed80  9#...U..E...q...

The first one (data packet) has broken checksum, the second one (control message) is valid. You can see this also in netstat:
root@something:~/ligato# netstat -n -s -u

(...)
Udp:
    1395047 packets received
    52500 packets to unknown port received
    164 packet receive errors
    2501579 packets sent
    0 receive buffer errors
    0 send buffer errors
    InCsumErrors: 164
(...)

Any ideas what I can try?

Best Regards,
BL

[nik-netlox]

Hi @bartekl,
Can you provide more information about your test?

1. How did you create the lb rule?
2. What is your topology?
3. Which tool are you using for the traffic?

[bartekl]

Hi,

1. It is as simple as:

loxicmd create lb A.C.45.83 --udp 500:500 --endpoints 100.65.0.1:1 --select persist
loxicmd create lb A.C.45.83 --udp 4500:4500 --endpoints 100.65.0.1:1 --select persist

2. All components are separate VMs.
   X (B.D.245.251) <---(1)---> (A.C.45.83) Y (private network) <---(2)---> (100.65.0.1) Z

• X, Z are Ubuntu 24.04 VMs w/ strongswan - these are configured to interconnect w/ IPSEC tunnel
• Y is loxilb standalone (docker w/ ipvlan interface facing public network, Ubuntu 24.04 VM
• (1) is in general L3 network (w/o any NAT or something in between, IP packets between X and Y are passed unchanged, modulo TTL ;-))
• (2) is a bit more complicated, since it consists of docker bridge (Y connects via veth to it), and actual transport between Y and Z is geneve tunnel, but I don't see this being relevant, since we're talking here about Z->X direction and I see that packet incoming to Y (loxilb) from Z is correct:

06:50:56.520251 02:42:cb:90:ff:a2 > 02:42:64:42:00:02, ethertype IPv4 (0x0800), length 146: (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto UDP (17), length 132)
    100.65.0.1.4500 > B.D.245.251.4500: [no cksum] UDP-encap: ESP(spi=0xc59daaee,seq=0x4), length 104

And this is correct behaviour of sender (Z) according to https://datatracker.ietf.org/doc/rfc3948/ (section 2.1).

3. Well, what loxilb is seeing here is IPsec's ESP in UDP encapsulation so the answer would be that linux kernel ipsec code (at Z) is the tool generating traffic, since encapsulated payload is "unreadable" (encrypted). I guess loxilb is not interested in anything past UDP header in this case anyway.

Best Regards, BL

[nik-netlox]

Hi @bartekl,
I tried but couldn't reproduce the issue. Checksum calculation is hardware offloaded. You can disable the checksum offloading on the interface and then try it.

ethtool -K <intf> tx-checksumming off

[TrekkieCoder]

Hi @bartekl Was your issue resolved ?

[bartekl]

Hi, sorry for late answer.

I ended up modifying this https://github.com/loxilb-io/loxilb-ebpf/blob/7bca92d858292dfe82ec45036f135dcda870ca9a/kprobe/loxilb-csum-kprobe.c to simply zero udp checksums (which is allowed by RFC mentioned earlier) and everything worked.
As of root cause it looks pretty similar (I'm using kernel that's supposed to have the fix) to this issue: https://patchwork.ozlabs.org/project/netdev/patch/20190912151353.2w7jakdrqljkfbsq@intra2net.com/, so I guess it's not a loxilb problem, but rather something inside kernel.

Anyway, thanks for the help.
Best Regards, BL