This repository contains the Golang source code for the Megalos CNI Plugin and the Megalos BGP Manager. Megalos CNI is compatible with Kubernetes v1.25+. Previous versions are not supported.
This plugin creates pure L2 LANs distributed across different worker nodes using VXLAN.
Before using the kathara-daemonset
, the Multus CNI must be deployed in the cluster.
See the official installation guide.
Once you have deployed the Multus CNI, you can deploy the kathara-daemonset
simply typing:
kubectl apply -f https://raw.githubusercontent.com/KatharaFramework/Megalos-CNI/master/kathara-daemonset.yml
Beware: Megalos CNI is used only for additional Pod interfaces created by Multus CNI! For the eth0
interface (required by Kubernetes) you must leverage on another CNI that manages L3 (e.g. Flannel, Calico).
NOTE: currently, Megalos CNI does not work on Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS) and ready-to-go cloud instances of Kubernetes, it only works on self-hosted clusters (since iptables
needs to install some custom rules).
This CNI creates a VXLAN network overlay over the Kubernetes cluster network. By default, VXLAN uses L3 multicast groups to deliver BUM traffic and to learn remote MAC Addresses, but multicast traffic is not always permitted inside a Kubernetes cluster network (see public clouds). To overcome this limitation, we disable the default VXLAN MAC Learning and we replace it with EVPN BGP, which ensures unicast traffic.
The BGP Manager works as follows:
- A DaemonSet is deployed on worker nodes, and each worker deploys a Pod that is a BGP speaker.
- A Service is created, with TCP port 179 and it is linked to a Deployment that contains a Pod that acts as a BGP Route Reflector. The Route Reflector accepts dynamic BGP peerings, so it does not require further configuration.
- Each worker node Pod has a single BGP peering with the BGP Route Reflector (using the Service Cluster IP Address).
When a Pod is created, the CNI works as follows:
- Each additional interface of the Pod is assigned to a specific VNI. Each VNI is associated to a different L2 LAN. In this way, we can distribute L2 LANs inside the cluster.
- On the worker node where the Pod is deployed, VTEPs for the specific Pod VNIs are created (if not already present).
- Connections between Pod interfaces and their speficic VTEPs are created using
veth
pairs.
At this point, the BGP control plane is able to fetch the information of Pod network interfaces. So, MAC Addresses of Pod interfaces are announced over BGP to the Route Reflector and reflected to all the other worker BGP speakers. Each worker BGP Pod receives the announce: if a VTEP for a specific VNI announced in the BGP message is present, the VTEP saves the association between MAC Address and the IP Address of the worker node where the new interfaces of that VNI are deployed.
In this repository you'll find two folders:
megalos-cni
: Golang CNI source code for Megalos.bgp-manager
: Dockerfile and Bash scripts to create thekathara/megalos-bgp-manager
Docker Image.
- Change the
IMAGE_NAME
variable inMakefile
with a custom tag<CUSTOM_NAME>
. - Run on terminal
make all
, this will:- Create a Docker container that will build the CNI binary from source for both
amd64
andarm64
architectures. - Push the Docker Image on your Docker Hub Repository.
- Create a Docker container that will build the CNI binary from source for both
- Change the
kathara-daemonset.yml
file and replacekathara/megalos-bgp-manager
with<CUSTOM_NAME>
. - Install the DaemonSet in your Kubernetes cluster using
kubeadm create -f kathara-daemonset.yml
.
It is possible to leverage on Megalos CNI as a standalone Multus CNI, in order to create pure L2 networks.
After following the Usage section, we can deploy an example network contained in a network.yaml
file:
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: test-network-1
spec:
config: '{
"cniVersion": "0.3.0",
"name": "net1",
"type": "megalos",
"suffix": "ntwork",
"vxlanId": 10
}'
The config
JSON contains the CNI configuration. The values are:
name
: a name for the network (maximum 4 characters)suffix
: used by Kathará to distinguish between networks with the samename
of different users (6 characters)vxlanId
: Megalos CNI relies on VXLAN, so you should assign a VXLAN ID manually (Kathará does it automatically)
At this point, we can create the network with the following command:
kubectl apply -f network.yaml
This configuration will create a VTEP, with VXLAN ID = 10, associated to the network net1-ntwork
.