Running eUPF inside the Kubernetes cluster with flannel

[HaoxinSEU]

Hi community,

Thanks for the nice eUPF project! We used the original UPF in Open5GS before and now we would like to shift to eUPF. We are trying to run it inside a K8s cluster, managed by flannel, but we meet some problems.

In our setup, we use srsRAN gNB, eUPF and other components from Open5GS. In the test, eUPF associated successfully with SMF, and the UE can connect to the network (AMF and SMF logs are fine, the UE showed the 5G logo). However, the UE had no traffic and cannot run ping or iperf3. We ran tcpdump on the eUPF pod, and it only got the PFCP packets, no traffic from/to the gNB.

We saw your example about running eUPF in k8s with Calico, so we are wondering if it can also work when the cluster is managed by flannel? Flannel encapsulates the traffic in the cluster, while Calico doesn't, will this make a difference for the functionality?

Any hints would be much appreciated!

Best regards,
Haoxin

Some details about our setup:
eUPF has a service, with port 2152 and 8805 forwarded;
Open5GS is deployed with the helm chart from Openverso;
If we use the original UPF from Open5GS, the UE connects without problem and can run iperf3.

Our configuration for eUPF is:

interface_name: [eth0]
xdp_attach_mode: generic
api_address: :8080
pfcp_address: 10.42.1.111:8805
pfcp_node_id: 10.43.130.186
metrics_address: :9090
n3_address: 10.42.1.111
gtp_peer:
echo_interval: 10
qer_map_size: 1024
far_map_size: 1024
pdr_map_size: 1024
resize_ebpf_maps: false
heartbeat_retries: 3
heartbeat_interval: 5
heartbeat_timeout: 5
logging_level: debug
feature_ueip: true
feature_ftup: true
ueip_pool: 10.45.0.0/16
teid_pool: 65536

[HaoxinSEU]

Here are the logs of eUPF, from which I cannot really tell if something is wrong...
console_log.txt
trace_log.txt

[pirog-spb]

Hi @HaoxinSEU

bpf_fib_lookup 10.45.0.13 -> 8.8.4.4: nexthop: 10.42.1.1

As I can see in trace log there are uplink GTP-U packet (N3), but no downlink packets (N6) processed by UPF.

Please verify your network configuration. Check that downlink packets are routed to UPF.

-- BR Alex

[pirog-spb]

Another possible problem - eUPF doesn't implemented NAT (while Open5GS provides NAT using linux masquerading).

If you need natting with eUPF you should use external NAT, currently.

[HaoxinSEU]

Hi Alex,

Thanks for the hints! We will look at these, and keep you updated.

Best,
Haoxin

[HaoxinSEU]

Hi @pirog-spb ,

We further dig into this issue. When the UE pings 8.8.8.8, we can see that the ping request is going out of the cluster via the virtual interface cni0, and then going out of the real Ethernet interface of the host (enp113s0). Here is the tcpdump result on the real interface of the host (doing the same experiment on the Open5GS UPF gives the same destination MAC address, so this DST MAC is not the problem).

hxsun@hxnuc:~$ sudo tcpdump -i enp113s0 -en icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp113s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:48:58.994731 48:21:0b:56:93:fa > 30:de:4b:a2:1a:5b, ethertype IPv4 (0x0800), length 98: 10.45.0.41 > 8.8.8.8: ICMP echo request, id 727, seq 1, length 64
11:49:01.838597 48:21:0b:56:93:fa > 30:de:4b:a2:1a:5b, ethertype IPv4 (0x0800), length 98: 10.45.0.41 > 8.8.8.8: ICMP echo request, id 728, seq 1, length 64
11:49:04.870603 48:21:0b:56:93:fa > 30:de:4b:a2:1a:5b, ethertype IPv4 (0x0800), length 98: 10.45.0.41 > 8.8.8.8: ICMP echo request, id 729, seq 1, length 64
11:49:07.910656 48:21:0b:56:93:fa > 30:de:4b:a2:1a:5b, ethertype IPv4 (0x0800), length 98: 10.45.0.41 > 8.8.8.8: ICMP echo request, id 730, seq 1, length 64

However, it seems that no reply is back to the host... Do you have any idea why this can happen?

Thanks again for your kind help!

Best,
Haoxin

[HaoxinSEU]

Hi Alex,

You are right! In the above tcpdump results we got, the source IP is the UE IP, which is wrong. The eUPF pod and the cluster should use NAT to change it.

We tried to clean the NAT table and add one in the eUPF pod:

iptables -D POSTROUTING 1 -t nat
iptables -t nat -A POSTROUTING -s 10.45.0.0/16 -o eth0 -j MASQUERADE

Now the NAT inside the eUPF pod looks like this:

root@eupf-test-4:/# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  10.45.0.0/16         anywhere  

However, we get the same results, and the source IP doesn't change... (still the UE IP). I'm wondering if we add the NAT rules wrongly?

Thank you so much for your help!

Best regards,
Haoxin

[HaoxinSEU]

Hi Alex,

On the host machine of the eUPF pod, after adding a NAT rule for 10.45.0.0/16, and a route via eUPF_POD_IP, the UE has the connectivity and can ping 8.8.8.8

But TBH this setup is a bit strange... maybe there are more elegant ways to do this...

[pirog-spb]

Hi Alex,

You are right! In the above tcpdump results we got, the source IP is the UE IP, which is wrong. The eUPF pod and the cluster should use NAT to change it.

We tried to clean the NAT table and add one in the eUPF pod:

iptables -D POSTROUTING 1 -t nat
iptables -t nat -A POSTROUTING -s 10.45.0.0/16 -o eth0 -j MASQUERADE

Now the NAT inside the eUPF pod looks like this:

root@eupf-test-4:/# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  10.45.0.0/16         anywhere  

However, we get the same results, and the source IP doesn't change... (still the UE IP). I'm wondering if we add the NAT rules wrongly?

Thank you so much for your help!

Best regards,
Haoxin

Configuring masquerading on eUPF pod is useless, because eUPF forwards packets via xdp hook without passing packets to linux network stack.

[pirog-spb]

Hi Alex,

On the host machine of the eUPF pod, after adding a NAT rule for 10.45.0.0/16, and a route via eUPF_POD_IP, the UE has the connectivity and can ping 8.8.8.8

But TBH this setup is a bit strange... maybe there are more elegant ways to do this...

We have examples of configuring routing to UE via BGP announces to K8s host. See open5gs-with-bgp deployment.

Another option is to use separete NAT pod. See this discussion topic #502

[HaoxinSEU]

This is very helpful! Thank you so much! It's very cool that eUPF also works with flannel in k8s

[HaoxinSEU]

We are doing some benchmark tests with this setup, the UE uplink throughput is reasonable, but the downlink throughput is low...

When using a 20MHz BW gNB and connecting one real UE (Samsung phone), in Speedtest, UE's DL throughput is 10 Mbps, UL is 20 Mbps. We notice that if we connect 2 UEs, then each of them will still get 10 Mbps DL, meaning the bottleneck is per session. So we are wondering if there is any limitation on the DL session bit rate? Thanks for any hints! :)

[pirog-spb]

Chech PDU session limitation. SMF may instruct UFP to limit bitrate. In the eUPF log you can see traces of PFCP messages from SMF with PDR/QER parameters.

[HaoxinSEU]

The log shows the limitation posed by SMF is 999999000 (I guess the unit is bps), and it should not be a bottleneck...

2024/06/19 09:04:17 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:999999000 MaxBitrateDL:999999000 StartUL:0 StartDL:0}

non_rt_pool#2-3078346 [010] d.s31 4048204.955327: bpf_trace_printk: upf: qer:1 gate_status:0 mbr:999999000