Adding a second dedicated network interface for longhorn replication

[sushyad]

I am trying to add a second network interface dedicated for longhorn replication using multus cni plugin together with ipvlan. Here is my PR from my fork to give you an idea what I am trying to do: #134

I was able to tweak the argocd recipe to:

• Bring up the second interface and assign it an ip address. I am able to ping the nodes on their secondary ip addresses from each other.
• Add ipvlan plugin to /opt/cni/bin folder on each node.
• Using this as a guide I was able to add multus configuration using argocd.

When I create a test pod wth two network interfaces it doesn't work and doesn't show the second interface as expected.

cat <<EOF | kubectl apply -f - 
apiVersion: v1
kind: Pod
metadata:
  name: app1
  annotations:
    k8s.v1.cni.cncf.io/networks: multus-conf
spec:
  containers:
  - name: app1
    command: ["/bin/sh", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
EOF

kubectl describe pod app1

gives

bash-5.2# kubectl describe pod app1 
Name:             app1
Namespace:        default
Priority:         0
Service Account:  default
Node:             metal0/192.168.0.115
Start Time:       Fri, 26 Jan 2024 19:49:49 +0000
Labels:           <none>
Annotations:      k8s.v1.cni.cncf.io/networks: multus-conf
Status:           Running
IP:               10.0.0.176
IPs:
  IP:  10.0.0.176
Containers:
......

instead of something lke this:

$ kubectl describe pod app1
Name:             app1
Namespace:        default
Priority:         0
Service Account:  default
Node:             node2/192.168.200.175
Start Time:       Fri, 11 Aug 2023 12:28:56 +0300
Labels:           <none>
Annotations:      k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "mynet",
                        "interface": "eth0",
                        "ips": [
                            "10.244.2.8"
                        ],
                        "mac": "86:69:28:4f:54:b3",
                        "default": true,
                        "dns": {},
                        "gateway": [
                            "10.244.2.1"
                        ]
                    },{
                        "name": "default/multus-conf",
                        "interface": "net1",
                        "ips": [
                            "192.168.200.100"
                        ],
                        "mac": "2a:1b:4d:89:66:c0",
                        "dns": {}
                    }]
                  k8s.v1.cni.cncf.io/networks: multus-conf
Status:           Running
IP:               10.244.2.8
IPs:
  IP:  10.244.2.8
Containers:
.....

Has anyone tried to do this before?

[khuedoan]

I don't have multiple NIC to reproduce but this is probably related cilium/cilium#23483

[khuedoan]

If this feature is important to you I think you can remove Cilium and use the default k3s CNI (Flannel), which seems to work with Multus

You can reference commits before 9f0d389 (install Cilium) and 65af4ff (remove MetalLB)

The disadvantage is that you may miss out on some future features that rely on eBPF.

[pandabear41]

I have reproduced this as well. Cilium features on paper are better, but they seem to lack for me vs Flannel or Calico.
I reverted back to default k3s CNI with PureLB (for now) with plans to test out Calico and their eBPF feature soon.

These three major issues I faced:

• Multus not working with Cilium CNI on both interfaces (cilium/cilium#23483). It works with Cilium CNI on one interface and other with Flannel, but this gets messy. The Cilium multi-homing feature (cilium/cilium#20129) has not been released yet.
• Pod-Pod VXLAN tunnels (using angelnu/pod-gateway) not working (cilium/cilium#22281) due to Cilium vulnerability (CVE-2019-13119). Fix has been not approved since 2022 CFP-2022-19-10: Advanced Routing rules for VPN use cases.
• Cilium IPv6 doesn't feel stable for what I want to use it for. I ran into many issues. Link-local IPv6 broken (cilium/cilium#26837). IPv6 only DNS issues. IPv6 only tunnelling doesn't work (cilium/cilium#17240).

[khuedoan]

IPv6 has a separate tracking issue #114

For this issue, I'm not sure if there's anything I can do on my end since I don't have or use multiple NICs. As far as I understand, there are two options:

• Replace Cilium with a different CNI that supports Multus
• Wait for upstream Cilium to support that feature: CFP: Cilium-native multi-homing cilium/cilium#20129

I'll leave this issue open for now in case someone has the same use case, but there's no action for it in this project.