Problem Using host-device CNI plugin

[infinitydon]

Hi,

I am trying to use host-device plugin for the N6 interface in the eUPF but once the upf process starts connectivity stops working..

Example multus config:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: n6-eupf1
spec:
  config: '{
            "cniVersion": "0.3.1",
            "plugins": [
                {
                    "type": "host-device",
                    "capabilities": { "ips": true },
                    "device": "eth2",
                    "ipam": {
                        "type": "static"
                    }
                }]
        }'

eUPF annotation:

podAnnotations:
  k8s.v1.cni.cncf.io/networks: >-
    [
    { "name": "n6-eupf1", "interface": "n6", "ips": ["192.168.101.3/24"] }    
    ]

Ping test to 192.168.101.3 fails and also when I ping from the eUPF pod it also fails.

Weird part also is that host interface eth2 will not be useable at all even if I delete the eUPF from the k8s cluster, the only time it can be used again is when I reboot the whole VM.

I tested with macvlan plugin and it works without any issues (the only issue is that native mode does not work with macvlan)

Any idea on what could be wrong?

[pirog-spb]

Hi Christopher @infinitydon,

First, I'd check that every thing is OK with K8s+multus. Could you exclude eth2(n6-eupf1) from eUPF configuration in order not to attach xdp program to it? Will connectivity be OK?

Also could you please enable debug logs(tracepipe) and check what's happening in ebpf xdp when you're experiencing connectivity loss?

-- BR, Alex

[infinitydon]

First, I'd check that every thing is OK with K8s+multus.

K8s+multus is okay, I double checked this by deploying open5gs UPF with the same set of Multus interfaces and ping connectivity works without any issues.

Could you exclude eth2(n6-eupf1) from eUPF configuration in order not to attach xdp program to it? Will connectivity be OK?

I tried, it's the same result, connectivity is lost once UE tries initiates a ping.

Also could you please enable debug logs(tracepipe) and check what's happening in ebpf xdp when you're experiencing connectivity loss?

Below are the debug logs, I was not able to find what could be wrong from the logs.

upf-debug-logs-dec-16.zip

[infinitydon]

I built a separate image with logging enabled.. So far it is not pointing to any issues:

 <...>-2461467 [006] d.s11 77744.707997: bpf_trace_printk: upf: gtp-u received
 <...>-2461467 [006] d.s11 77744.708011: bpf_trace_printk: SDF: filter protocol: 4
 <...>-2461467 [006] d.s11 77744.708013: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
 <...>-2461467 [006] d.s11 77744.708014: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
 <...>-2461467 [006] d.s11 77744.708036: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
nr-gnb-2461467 [006] d.s11 77744.708037: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
nr-gnb-2461467 [006] d.s11 77744.708038: bpf_trace_printk: SDF: packet protocol: 0
nr-gnb-2461467 [006] d.s11 77744.708040: bpf_trace_printk: SDF: packet source ip: 10.11.0.2, destination ip: 192.168.101.1
nr-gnb-2461467 [006] d.s11 77744.708041: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
nr-gnb-2461467 [006] d.s11 77744.708042: bpf_trace_printk: upf: sdf filter doesn't match teid:2
nr-gnb-2461467 [006] d.s11 77744.708044: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
nr-gnb-2461467 [006] d.s11 77744.708046: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:999999000
nr-gnb-2461467 [006] d.s11 77744.708047: bpf_trace_printk: upf: session for teid:2 far:1 outer_header_removal:0
nr-gnb-2461467 [006] d.s11 77744.708071: bpf_trace_printk: upf: bpf_fib_lookup 10.11.0.2 -> 192.168.101.1: nexthop: 192.168.101.1
<idle>-0       [027] d.s.1 77744.992074: bpf_trace_printk: upf: no downlink session for ip:255.255.255.255
<idle>-0       [001] d.s.1 77744.992084: bpf_trace_printk: upf: no downlink session for ip:255.255.255.255
belite-10565   [012] d.s11 77745.475873: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10565   [012] d.s11 77745.475952: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10555   [007] d.s11 77745.476014: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10533   [031] d.s11 77745.476061: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10555   [007] d.s11 77745.476097: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10562   [027] d.s11 77745.476118: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10533   [031] d.s11 77745.476156: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10561   [010] d.s11 77745.476178: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10562   [027] d.s11 77745.476180: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
  bird-2451733 [024] d.s11 77745.476276: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10533   [031] d.s11 77745.476319: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10555   [007] d.s11 77745.476342: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
belite-10545   [030] d.s11 77745.476362: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
  bird-2451733 [024] d.s11 77745.476417: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
  eupf-2451761 [029] d.s11 77745.476647: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
  eupf-2451711 [015] d.s11 77745.476701: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49
nr-gnb-2461467 [006] d.s11 77745.708546: bpf_trace_printk: upf: gtp-u received
nr-gnb-2461467 [006] d.s11 77745.708562: bpf_trace_printk: SDF: filter protocol: 4
nr-gnb-2461467 [006] d.s11 77745.708564: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
nr-gnb-2461467 [006] d.s11 77745.708565: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
nr-gnb-2461467 [006] d.s11 77745.708581: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
nr-gnb-2461467 [006] d.s11 77745.708582: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
nr-gnb-2461467 [006] d.s11 77745.708583: bpf_trace_printk: SDF: packet protocol: 0
nr-gnb-2461467 [006] d.s11 77745.708584: bpf_trace_printk: SDF: packet source ip: 10.11.0.2, destination ip: 192.168.101.1
nr-gnb-2461467 [006] d.s11 77745.708585: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
nr-gnb-2461467 [006] d.s11 77745.708586: bpf_trace_printk: upf: sdf filter doesn't match teid:2
nr-gnb-2461467 [006] d.s11 77745.708587: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
nr-gnb-2461467 [006] d.s11 77745.708626: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:999999000
nr-gnb-2461467 [006] d.s11 77745.708628: bpf_trace_printk: upf: session for teid:2 far:1 outer_header_removal:0
nr-gnb-2461467 [006] d.s11 77745.708631: bpf_trace_printk: upf: bpf_fib_lookup 10.11.0.2 -> 192.168.101.1: cached ifindex: 17
s-smfd-2461289 [027] d.s11 77746.084286: bpf_trace_printk: upf: no downlink session for ip:10.1.29.21
<idle>-0       [001] d.s.1 77746.633840: bpf_trace_printk: upf: no downlink session for ip:192.168.101.4
nr-gnb-2461467 [006] d.s11 77746.708593: bpf_trace_printk: upf: gtp-u received
nr-gnb-2461467 [006] d.s11 77746.708607: bpf_trace_printk: SDF: filter protocol: 4
nr-gnb-2461467 [006] d.s11 77746.708609: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
nr-gnb-2461467 [006] d.s11 77746.708610: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
nr-gnb-2461467 [006] d.s11 77746.708611: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
nr-gnb-2461467 [006] d.s11 77746.708612: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
nr-gnb-2461467 [006] d.s11 77746.708613: bpf_trace_printk: SDF: packet protocol: 0
nr-gnb-2461467 [006] d.s11 77746.708614: bpf_trace_printk: SDF: packet source ip: 10.11.0.2, destination ip: 192.168.101.1
nr-gnb-2461467 [006] d.s11 77746.708615: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
nr-gnb-2461467 [006] d.s11 77746.708616: bpf_trace_printk: upf: sdf filter doesn't match teid:2
nr-gnb-2461467 [006] d.s11 77746.708617: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
nr-gnb-2461467 [006] d.s11 77746.708618: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:999999000
nr-gnb-2461467 [006] d.s11 77746.708619: bpf_trace_printk: upf: session for teid:2 far:1 outer_header_removal:0
nr-gnb-2461467 [006] d.s11 77746.708622: bpf_trace_printk: upf: bpf_fib_lookup 10.11.0.2 -> 192.168.101.1: cached ifindex: 17
<idle>-0       [027] d.s.1 77748.460111: bpf_trace_printk: upf: no downlink session for ip:192.168.101.3
s-smfd-2461289 [027] d.s11 77748.800897: bpf_trace_printk: upf: no downlink session for ip:10.1.29.49

I switched to vmxnet3 from virtio_pci, still the same problem persists

But I also noticed that the packet drop count is increasing on the N6 (n6eupf1) interface TX:

Interface statistics before sending 3 ping packets from UE1:

/ # ip -s link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    RX:  bytes packets errors dropped  missed   mcast
             0       0      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
             0       0      0       0       0       0
3: eth0@if57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 xdpgeneric qdisc noqueue state UP mode DEFAULT group default
    link/ether 52:09:b1:fb:83:e9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    prog/xdp id 469 name upf_ip_entrypoi tag 95500ea13eb6efb5 jited
    RX:  bytes packets errors dropped  missed   mcast
       4179643    1980      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
         88292    1255      0       1       0       0
17: n6eupf1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether bc:24:11:89:21:b0 brd ff:ff:ff:ff:ff:ff
    alias eth2
    prog/xdp id 469 name upf_ip_entrypoi tag 95500ea13eb6efb5 jited
    RX:  bytes packets errors dropped  missed   mcast
        113699    1141      0   73550       0       0
    TX:  bytes packets errors dropped carrier collsns
          8487     112      0      57       0       0
    altname enp0s20
    altname ens20

Interface statistics after sending 3 ping packets from UE1:

/ # ip -s link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    RX:  bytes packets errors dropped  missed   mcast
             0       0      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
             0       0      0       0       0       0
3: eth0@if57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 xdpgeneric qdisc noqueue state UP mode DEFAULT group default
    link/ether 52:09:b1:fb:83:e9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    prog/xdp id 469 name upf_ip_entrypoi tag 95500ea13eb6efb5 jited
    RX:  bytes packets errors dropped  missed   mcast
       4190041    2133      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
         94238    1344      0       1       0       0
17: n6eupf1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether bc:24:11:89:21:b0 brd ff:ff:ff:ff:ff:ff
    alias eth2
    prog/xdp id 469 name upf_ip_entrypoi tag 95500ea13eb6efb5 jited
    RX:  bytes packets errors dropped  missed   mcast
        124305    1246      0   73630       0       0
    TX:  bytes packets errors dropped carrier collsns
          9176     121      0      60       0       0
    altname enp0s20
    altname ens20

So it looks like it is the eUPF that is dropping the packets? Can you suggest where to check further?

[infinitydon]

@pirog-spb - Can you kindly suggest how to troubleshoot/solve the packet drop as per my recent observation.

Thanks

[pirog-spb]

@infinitydon I see no downlink packets in log for your test traffic flow.

This flow was forwarded to N6

bpf_fib_lookup 10.11.0.2 -> 192.168.101.1: nexthop: 192.168.101.1

But seems like no incoming packets 192.168.101.1 -> 10.11.0.2 are there. Are you sure that these packets should be there? Can you capture pcap dump on the corresponding interface?

Considering packet drops, I suppose that packets are dropped on RX. So eUPF even don't see any packets. May be there is some issue with checksums?

RX:  bytes packets errors dropped  missed   mcast
        124305    1246      0   73630       0       0

[infinitydon]

@pirog-spb - I have tried to set checksum off, none is working..

The only mode that works is macvlan CNI plugin (generic mode).

I have tried Native mode for veth, virtio_net, vmxnet3, they are not working and I am certain now that it's the eUPF dropping the packets because using those mode for open5gs UPF works.

For now I am not able to proceed with testing the native mode.

Have you tested native mode running in Kubernetes? If yes let me know your working configuration.

[pirog-spb]

Yes, we have tested native mode in K8s. And it's little bit tricky.

For veth drive in order to load ebpf program in native mode inside a pod you have to load ebpf program (any) on corresponding veth paired interface on the host. See #116 for details. You can use zero_entrypoint_bpf program from eupf container.

In order to use virtio_net, vmxnet3 with native xdp mode some hypervisor options have to be tuned. See discussion #507

But. I think that your initial problem is not in the attachement mode. I'd rather investigate rx drops. You can see detailed info about drop reason using tools like ethtool or netstat. See #170 for details

[infinitydon]

@pirog-spb - All the tools you referenced did not help, netstat was not showing any errors, ethtool features is broad, I already tried to disable checksuming, it did not help.

Eventually I decided to use virtio_net interfaces for the gNB N3, eupf N3, N6, with that at least there should be no changes in the interface features, but there is slight improvement in the iperf3 stats with the Native mode.

bash-5.1# iperf3 -c 192.168.101.254 -B 10.11.0.2
Connecting to host 192.168.101.254, port 5201
[  5] local 10.11.0.2 port 43111 connected to 192.168.101.254 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  46.5 MBytes   390 Mbits/sec    0   2.34 MBytes
[  5]   1.00-2.00   sec  47.5 MBytes   398 Mbits/sec  993   1.57 MBytes
[  5]   2.00-3.00   sec  43.8 MBytes   367 Mbits/sec    0   1.67 MBytes
[  5]   3.00-4.00   sec  50.0 MBytes   419 Mbits/sec    7   1.23 MBytes
[  5]   4.00-5.00   sec  50.0 MBytes   419 Mbits/sec    0   1.31 MBytes
[  5]   5.00-6.00   sec  50.0 MBytes   419 Mbits/sec    0   1.37 MBytes
[  5]   6.00-7.00   sec  48.8 MBytes   409 Mbits/sec    0   1.41 MBytes
[  5]   7.00-8.00   sec  41.2 MBytes   346 Mbits/sec    0   1.44 MBytes
[  5]   8.00-9.00   sec  51.2 MBytes   430 Mbits/sec    0   1.46 MBytes
[  5]   9.00-10.00  sec  47.5 MBytes   398 Mbits/sec  592    748 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   476 MBytes   400 Mbits/sec  1592             sender
[  5]   0.00-10.01  sec   474 MBytes   397 Mbits/sec                  receiver

Compared to the generic mode:

bash-5.1# iperf3 -c 192.168.101.254 -B 10.11.0.2
Connecting to host 192.168.101.254, port 5201
[  5] local 10.11.0.2 port 47325 connected to 192.168.101.254 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  44.6 MBytes   374 Mbits/sec    0   2.08 MBytes
[  5]   1.00-2.00   sec  46.2 MBytes   388 Mbits/sec  1009   1.54 MBytes
[  5]   2.00-3.00   sec  43.8 MBytes   367 Mbits/sec    0   1.64 MBytes
[  5]   3.00-4.00   sec  40.0 MBytes   336 Mbits/sec    0   1.71 MBytes
[  5]   4.00-5.00   sec  43.8 MBytes   367 Mbits/sec    0   1.77 MBytes
[  5]   5.00-6.00   sec  48.8 MBytes   409 Mbits/sec    0   1.81 MBytes
[  5]   6.00-7.00   sec  48.8 MBytes   409 Mbits/sec    0   1.84 MBytes
[  5]   7.00-8.00   sec  46.2 MBytes   388 Mbits/sec    0   1.86 MBytes
[  5]   8.00-9.00   sec  43.8 MBytes   367 Mbits/sec  1383    983 KBytes
[  5]   9.00-10.00  sec  38.8 MBytes   325 Mbits/sec  922    242 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   445 MBytes   373 Mbits/sec  3314             sender
[  5]   0.00-10.00  sec   442 MBytes   370 Mbits/sec                  receiver

For now I think I will stick to using macvlan CNI plugin.

One issue I encountered, not sure if it is a bug, I configured the eUPF to use a different N3 interface IP:

env:
  UPF_PFCP_NODE_ID: $(MY_POD_IP)
  UPF_N3_ADDRESS: 192.168.102.1

command: ["/bin/sh", "-c"]

args: [
   mkdir -p /etc/iproute2/;
   echo '1000 n6if' >> /etc/iproute2/rt_tables;
   ip rule add from 10.11.0.0/16 table n6if;
   ip route add default via 192.168.101.1 dev n6eupf1 table n6if;
   sh /app/bin/entrypoint.sh --config /app/conf/config.yml
 ]
configMaps:
  config:
    data:
      config.yml: |
        interface_name: [eth0, n3eupf1, n6eupf1]
        api_address: :8080
        pfcp_address: :8805
        metrics_address: :9090
        xdp_attach_mode: generic

podAnnotations:
  k8s.v1.cni.cncf.io/networks: >-
    [
    { "name": "n3-eupf1", "interface": "n3eupf1", "ips": ["192.168.102.1/24"] },
    { "name": "n6-network", "interface": "n6eupf1", "ips": ["192.168.101.3/24"] }    
    ]

The eUPF did not use the N3 IP for the GTP tunnel instead it kept using the UPF_PFCP_NODE_ID i.e. the N4 IP. I then tried setting the UPF_PFCP_NODE_ID to the N3 IP to see if it will change, it didn't, the only time it changed was when the SMF N4 communication did not come through eth0, as long as the SMF is initiating PFCP communication via the eth0, the eUPF will not use the N3 IP address that is configured. Maybe you can check and verify also.

[s5uishida]

@infinitydon

One issue I encountered, not sure if it is a bug, I configured the eUPF to use a different N3 interface IP:

Have you tried the following?

feature_ftup: true

[pirog-spb]

@infinitydon Could you please clarify what's wrong with N3 IP? Configured N3 IP is not set as a source IP of GTP packets towards gNB?

Considering your perf test - keep in mind that eupf may not be a bottleneck. If you used UERANSIM, check CPU usage when test is running.

BTW

interface_name: [eth0, n3eupf1, n6eupf1]

You don't have to configure interfaces if no data plane packets expected there. Seems like you may exclude eth0 from configuration.

[infinitydon]

@pirog-spb Yes, Configured N3 IP is not set as a source IP of GTP packets towards gNB

Sample config:

env:
  UPF_PFCP_NODE_ID: $(MY_POD_IP)
  UPF_N3_ADDRESS: 192.168.102.1

command: ["/bin/sh", "-c"]

args: [
   mkdir -p /etc/iproute2/;
   echo '1000 n6if' >> /etc/iproute2/rt_tables;
   ip rule add from 10.11.0.0/16 table n6if;
   ip route add default via 192.168.101.1 dev n6eupf1 table n6if;
   sh /app/bin/entrypoint.sh --config /app/conf/config.yml
 ]

configMaps:
  config:
    data:
      config.yml: |
        interface_name: [n3eupf1, n6eupf1]
        api_address: :8080
        pfcp_address: :8805
        metrics_address: :9090
        xdp_attach_mode: native

podAnnotations:
  k8s.v1.cni.cncf.io/networks: >-
    [
    { "name": "n3-eupf1", "interface": "n3eupf1", "ips": ["192.168.102.1/24"] },
    { "name": "n6-eupf1", "interface": "n6eupf1", "ips": ["192.168.101.3/24"] }
    ]

PCAP showed that GTP tunnel was not using 192.168.102.1 instead it was using the k8s eth0 IP:

15:00:12.629317 cali3b4c2ba40f0 In  IP 192.168.102.3.2152 > 10.1.29.29.2152: UDP, length 100
15:00:13.629578 cali3b4c2ba40f0 In  IP 192.168.102.3.2152 > 10.1.29.29.2152: UDP, length 100
15:00:14.629603 cali3b4c2ba40f0 In  IP 192.168.102.3.2152 > 10.1.29.29.2152: UDP, length 100

As shown in the above config, removing eth0 from the XDP interface configuration did not fix it.

@s5uishida suggestion worked, I configured it as below:

env:
  UPF_PFCP_NODE_ID: $(MY_POD_IP)
  UPF_N3_ADDRESS: 192.168.102.1

command: ["/bin/sh", "-c"]

args: [
   mkdir -p /etc/iproute2/;
   echo '1000 n6if' >> /etc/iproute2/rt_tables;
   ip rule add from 10.11.0.0/16 table n6if;
   ip route add default via 192.168.101.1 dev n6eupf1 table n6if;
   sh /app/bin/entrypoint.sh --config /app/conf/config.yml
 ]
configMaps:
  config:
    data:
      config.yml: |
        interface_name: [eth0, n3eupf1, n6eupf1]
        api_address: :8080
        pfcp_address: :8805
        metrics_address: :9090
        feature_ftup: true
        teid_pool: 65536
        xdp_attach_mode: native

podAnnotations:
  k8s.v1.cni.cncf.io/networks: >-
    [
    { "name": "n3-eupf1", "interface": "n3eupf1", "ips": ["192.168.102.1/24"] },
    { "name": "n6-eupf1", "interface": "n6eupf1", "ips": ["192.168.101.3/24"] }    
    ]

PCAP trace:

10:14:54.662106 tap126i2 Out IP 192.168.102.3.2152 > 192.168.102.1.2152: UDP, length 100
10:14:54.662570 tap126i7 P   IP 192.168.102.1.2152 > 192.168.102.3.2152: UDP, length 100
10:14:54.662593 fwln126i7 Out IP 192.168.102.1.2152 > 192.168.102.3.2152: UDP, length 100
10:14:54.662609 fwpr126p7 P   IP 192.168.102.1.2152 > 192.168.102.3.2152: UDP, length 100
10:14:54.662612 fwpr126p6 Out IP 192.168.102.1.2152 > 192.168.102.3.2152: UDP, length 100
10:14:54.662613 fwln126i6 P   IP 192.168.102.1.2152 > 192.168.102.3.2152: UDP, length 100
10:14:54.662615 tap126i6 Out IP 192.168.102.1.2152 > 192.168.102.3.2152: UDP, length 100
10:14:55.662225 tap126i6 P   IP 192.168.102.3.2152 > 192.168.102.1.2152: UDP, length 100