Disclaimer: This is NOT an official Cisco application and comes with absolute NO WARRANTY!
Please check the LICENSE for further information.
Helm chart repository and instructions here.
- Kubernetes v1.16+ (Tested with ACI 4.1, 4.2 and 5.0)
- Openshift Container Platform v4.3+ (Should work but never tested)
Note: The operator uses Volumes. It will NOT start if docker-novolume-plugin is used.
Note: This operator now uses apiextensions.k8s.io/v1beta1 to manage CustomResourceDefinitions.
If you need to run the RMLAB ACI Operator on Kubernetes <1.16 or Openshift 3.x, please use the v1alpha branch. However, keep in mind that not all features are available on that release.
| Topic | Kubernetes | Openshift |
|---|---|---|
| Resource creation | Creates a namespace | Creates a project |
| Runs on namespace | kube-default | aci-containers-system |
- The operator now supports multiple use cases to accommodate different business needs
- Controller doesn't have affinity anymore, it will scheduled on any available worker node
- Operator now complies with Operator Framework v1
- Finalizers are now supported for resource cleanup
- Kubectl now shows consumer/provider contracts for the EPG your namespace is attached to. These values are updated by default every 10 seconds
- Option to detach the operator in case at some point in time you want to go manual
- Support for non default AP and Bridge Domain
- Helm installer
helm repo add rmlab-aci-operator https://rtortori.github.io/rmlab-aci-operator-helm/
helm install aci-op rmlab-aci-operator/latest
Or you can install manually by cloning this repo and run:
installers/acins-kube-installer.sh
helm repo add rmlab-aci-operator https://rtortori.github.io/rmlab-aci-operator-helm/
helm install aci-op rmlab-aci-operator/latest --set global.containerPlatform=openshift
Or you can install manually by cloning this repo and run:
installers/acins-ocp-installer.sh
helm unistall aci-op
Or you can uninstall manually by cloning this repo and run:
installers/acins-kube-uninstaller.sh
helm unistall aci-op
Or you can uninstall manually by cloning this repo and run:
installers/acins-ocp-uninstaller.sh
Compared to previous versions, from version 1 the RMLAB ACI Kubernetes Operator has been reworked to address three challenges:
- Be compliant with the v1 Operator SDK framework data structure and features
- Implement multiple ACI CNI use cases
- Allow the user to detach the operator in case manual (traditional) control is required at some point in time (e.g. you realize the infrastructure is getting too complex to be managed by the operator)
For further information on SDK framework v1, please click here
The acinamespaces.rmlab.cisco.com CRD implements a resource called AciNamespace under the API rmlab.cisco.com/v1.
While previous versions of this operator only supported a single use case (create an EPG and annotate a namespace), since version 1 the operator supports multiple use cases which can be grouped into two strategies:
- Provision an ACI EPG and inherit the contracts from an EPG contract master. Create or update a namespace with ACI annotation (Use cases 1, 3 and 8. Described below)
- Attach to an existing EPG. Create or update a namespace with ACI annotation (Use cases 2, 4 and 6. Described below).
When using strategy no.1. ACI Admins are required to pre-provision EPGs with the right contract as per corporate policies.
Once the EPGs have been defined, Kubernetes admins can reference to their names as EPG Contract Masters (created EPGs will inherit contracts from those EPGs).
The following table describes the operator use cases. Each use case is basically a combination of the specs you can configure in the CR.
epgcontractmaster and op_managed are specs of the Custom Resource you create (CR).
| Use Case # | Use Case Description | Existing EPG | epgcontractmaster | op_managed | Notes |
|---|---|---|---|---|---|
| 1 | Create a new EPG from an EPG contract master and fully manage the EPG and Namespace. When the AciNamespace is deleted, the EPG and the Namespace will also be deleted | NO | explicit | True | - |
| 2 | The EPG will be adopted by the operator, attached to the VMM domain and attached to an EPG contract master. When the AciNamespace is deleted, the EPG and the Namespace will also be deleted | YES | explicit | True | WARNING: You might delete an existing EPG when you remove the CR, ensure this is what you want |
| 3 | Create a new EPG from an EPG contract master and fully manage the EPG and Namespace. When the AciNamespace is deleted, the operator is detached and EPG and namespaces will go in manual mode | NO | explicit | False | - |
| 4 | The EPG will be adopted by the operator, a bind to the VMM domain will be added and attached to an EPG contract master. When the AciNamespace is deleted, the operator is detached and EPG and namespaces will go in manual mode | YES | explicit | False | - |
| 5 | NOT SUPPORTED | NO | not specified in the CR | True | There's no point to create a new EPG with no contracts attached, PODs in this namespace will not work. For this use case, the operator will do NOTHING |
| 6 | The EPG will be adopted by the operator and attached to the VMM domain. When the AciNamespace is deleted, the EPG and the Namespace will also be deleted | YES | not specified in the CR | True | WARNING: You might delete an existing EPG when you remove the CR, ensure this is what you want |
| 7 | NOT SUPPORTED | NO | not specified in the CR | False | There's no point to create a new EPG with no contracts attached, PODs in this namespace will not work. For this use case, the operator will do NOTHING |
| 8 | Create and annotate a Namespace to an existing EPG, adding a VMM binding to it. The operator will still ensure the EPG always exists to be consistent with the Namespace annotation, however, when the AciNamespace is deleted, the operator is detached and both EPG and Namespace will go in manual mode | NO | not specified in the CR | True | - |
The following is a sample Custom Resource (CR) for an AciNamespace.
apiVersion: rmlab.cisco.com/v1
kind: AciNamespace
metadata:
name: new-epg-uc1
spec:
epgcontractmaster: "kube-default"
openshiftproject: False
op_managed: True
The following table describes the AciNamespaces Custom Resource (CR) specs.
| Spec | Description | Example | Optional/Mandatory | Default |
|---|---|---|---|---|
| epgcontractmaster | The EPG Master Contract you want to attach your EPG to | "kube-default" | Optional | "N/A" |
| openshiftproject | If True, the CR will create/update an Openshift Project | False | Optional | False |
| op_managed | If False, when you delete the CR the operator will detach to the namespace and EPG, allowing you to manually manage the infrastructure | True | Mandatory | True |
| applicationprofile | The ACI AP | "kubernetes" | Optional | "kubernetes" |
| bridgedomain | The ACI Bridge Domain | "kube-pod-bd" | Optional | "kube-pod-bd" |
The folder config/samples contains CR examples for all use cases.
This is an example for Use Case #1:
apiVersion: rmlab.cisco.com/v1
kind: AciNamespace
metadata:
name: frontend
spec:
epgcontractmaster: "kube-default"
applicationprofile: "kubernetes"
bridgedomain: "kube-pod-bd"
openshiftproject: False
op_managed: True
Where name is the desired name of the EPG and Kubernetes namespace
This will:
- Create a new
AciNamespaceresource namedfrontend - Create a new EPG in the
kubernetesapplication profile - Bind the
frontendEPG to the VMM domain - Configure
kube-defaultEPG as the contract master for thefrontendEPG - Creates or updates a Kubernetes namespace called
frontendwith the correct opflex annotation
cat <<EOF | kubectl apply -f -
apiVersion: rmlab.cisco.com/v1
kind: AciNamespace
metadata:
name: frontend
spec:
epgcontractmaster: "kube-default"
applicationprofile: "kubernetes"
bridgedomain: "kube-pod-bd"
openshiftproject: False
op_managed: True
EOF
acinamespace.rmlab.cisco.com/frontend created
$ kubectl get acinamespace
NAME EPG-CONTRACT-MASTER ACI-BD MANAGED
frontend kube-default kube-pod-bd true
You can fetch more details, like the list of provider and consumer contracts your EPG is using with the "-o wide" option
$ k get acinamespace -o wide
NAME EPG-CONTRACT-MASTER ACI-BD M-EPG-C-CONTRACTS M-EPG-P-CONTRACTS C-CONTRACTS P-CONTRACTS MANAGED
frontend kube-default kube-pod-bd [['uni/tn-pvt20/brc-dns', 'uni/tn-pvt20/brc-kube-api', 'uni/tn-pvt20/brc-icmp']] [['uni/tn-pvt20/brc-health-check']] [] [] true
or using kubectl describe acinamespace
$ kubectl describe aci frontend
Name: frontend
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"rmlab.cisco.com/v1","kind":"AciNamespace","metadata":{"annotations":{},"name":"frontend","namespace":"default"},"spec":{"ap...
API Version: rmlab.cisco.com/v1
Kind: AciNamespace
Metadata:
Creation Timestamp: 2020-10-17T22:01:09Z
Finalizers:
finalizer.aci.rmlab.cisco.com
Generation: 1
Resource Version: 1317176
Self Link: /apis/rmlab.cisco.com/v1/namespaces/default/acinamespaces/frontend
UID: ab7adda3-1313-4266-82d6-d9843ec15831
Spec:
Applicationprofile: kubernetes
Bridgedomain: kube-pod-bd
Epgcontractmaster: kube-default
op_managed: true
Openshiftproject: false
Status:
bridge_domain: kube-pod-bd
consumer_contracts: []
mepg_consumer_contracts: [['uni/tn-pvt20/brc-dns', 'uni/tn-pvt20/brc-kube-api', 'uni/tn-pvt20/brc-icmp']]
mepg_dn: ['uni/tn-pvt20/ap-kubernetes/epg-kube-default']
mepg_provider_contracts: [['uni/tn-pvt20/brc-health-check']]
provider_contracts: []
Events: <none>
This is how the EPG looks in APIC
You can switch mode (Operator managed to unmanaged and viceversa) patching a specific AciNamespace at any time
kubectl patch aci youracinamespace -p '{"spec":{"op_managed":false}}' --type=merge
Once you delete the CR, if you wish to retake control, you need to recreate the CR.
The operator respects the data structure of the Operator Framework SDK v1.
Please have a look here for the Ansible operator quick start.
To modify the AciNamespace CustomResourceDefinition, edit the file config/bases/rmlab.cisco.com_acinamespaces.yaml
To modify the controller for Kubernetes (e.g. use your own image), edit the file config/manager-k8s
For Openshift, edit the file config/manager-openshift
You don't need to modify anything in the manifests directory, the manifest files will be created automatically when you build.
The playbooks directory contains the main playbook and the finalizer. The playbooks contain descriptions for each task to describe the controller logic.
Modify the requirements.yaml file to add more collections to the Ansible controller, in case you want to extend the operator and need additional modules.
The watches.yaml file specifies the Kubernetes APIs the controller needs to watch, the playbook it should run, the finalizer that should use when AciNamespaces are terminated.
You can customize the reconcilePeriod if you feel 10 seconds is too aggressive.
Kubernetes (modify the IMG variable with your controller image)
make k8s-installer && make k8s-uninstaller && make docker-build docker-push IMG=your/image:tag
Openshift (modify the IMG variable with your controller image)
make ocp-installer && make ocp-uninstaller && make docker-build docker-push IMG=IMG=your/image:tag