Add Kube-Burner to Tools Directory for ArgoCD Benchmarking

README.md

@@ -2,12 +2,16 @@
 
 ## Current Tools
 
-### ArgoCD/ArgoWorkflows tooling based on Amazon EKS - [Link](tools/awsbenchmarking-nodejs)
+### 1.  ArgoCD/ArgoWorkflows tooling based on Amazon EKS - [Link](tools/awsbenchmarking-nodejs)
 This tooling is built on nodejs and utilizes various AWS tools to deploy and run benchmarks for both ArgoCD and ArgoWorkflows on Amazon EKS. This is the same tooling utilized to capture data for the following blog posts/talks:
 
 * [Argo CD Application Controller Scalability Testing on Amazon EKS](https://aws.amazon.com/blogs/opensource/argo-cd-application-controller-scalability-testing-on-amazon-eks/)
 * [Argo CD Benchmarking - Pushing the Limits and Sharding Deep Dive](https://cnoe.io/blog/argo-cd-application-scalability)
 * [Argo Workflows Controller Scalability Testing on Amazon EKS](https://cnoe.io/blog/argo-workflow-scalability)
 * [Adobe/AWS: Key Takeaways from Scaling Adobe's CI/CD Solution to Support 50K Argo CD Apps ](https://www.youtube.com/watch?v=7yVXMCX62tY)
 
+### 2.  Kube-Burner - [Link](tools/kube-burner)
+
+[Kube-Burner](https://kube-burner.github.io/kube-burner/latest/) is a resource creation and metrics collection tool. It is used to test ArgoCD performance by creating multiple ArgoCD Applications and collecting performance metrics such as CPU, memory, and API server requests. For detailed setup and usage, check the [Kube-Burner README](tools/kube-burner/readme.md).
+
 

tools/kube-burner/config.yaml

@@ -0,0 +1,48 @@
+# Global configuration
+global:
+  gc: false
+
+# Jobs configuration
+jobs:
+  # Job to create Namespaces, Roles, and RoleBindings
+  - name: create-namespaces
+    jobIterations: 100
+    cleanup: false
+    qps: 10  # Increased qps to handle more requests
+    burst: 50 # Increased burst to allow higher initial throughput
+    objects:
+      - kind: Namespace
+        objectTemplate: template/namespace.yaml
+        replicas: 1
+  - name: create-role
+    jobIterations: 100
+    cleanup: false
+    qps: 10
+    burst: 50
+    objects:
+      - kind: Role
+        objectTemplate: template/role.yaml
+        replicas: 1
+  - name: create-rolebinding
+    jobIterations: 100
+    cleanup: false
+    qps: 10
+    burst: 50
+    objects:
+      - kind: RoleBinding
+        objectTemplate: template/rolebinding.yaml
+        replicas: 1            
+  # Job to create ArgoCD applications
+  - name: create-applications
+    jobIterations: 100
+    cleanup: false
+    waitWhenFinished: true
+    qps: 10
+    burst: 50
+    objects:
+      - kind: Application
+        objectTemplate: template/application.yaml
+        replicas: 1
+        waitOptions:
+          forCondition: "Synced"
+          customStatusPath: ".operationState.syncResult.resources[].status"

tools/kube-burner/metric/ep.yaml

@@ -0,0 +1,7 @@
+- endpoint: <metric-endpoint>
+  token: ""
+  metrics:
+  - metric/metrics-profile.yaml
+  indexer:
+    type: local
+    metricsDirectory: my-metrics

tools/kube-burner/metric/metrics-profile.yaml

@@ -0,0 +1,34 @@
+# Metrics Profile for ArgoCD Performance Testing
+
+# CPU Usage of ArgoCD Application Controller
+- query: irate(process_cpu_seconds_total{container="argocd-application-controller",namespace=~".+"}[1m])
+  metricName: argocdAppControllerCPU
+
+# Memory Usage of ArgoCD Application Controller
+- query: go_memstats_heap_alloc_bytes{container="argocd-application-controller",namespace=~".+"}
+  metricName: argocdAppControllerHeapAllocMemory
+
+- query: go_memstats_heap_inuse_bytes{container="argocd-application-controller",namespace=~".+"}
+  metricName: argocdAppControllerHeapInuseMemory
+
+
+# Number of k8s resource objects in the cache
+- query: sum(argocd_cluster_api_resource_objects{namespace=~".+"})
+  metricName: argocdClusterApiResourceObjects
+
+# Number of monitored Kubernetes API resources
+- query: sum(argocd_cluster_api_resources{namespace=~".+"}) 
+  metricName: argocdClusterApiResources
+
+# Total IO Operations of ArgoCD Application Controller
+- query: sum(rate(container_fs_reads_total{pod=~"openshift-gitops-application-controller-.*",namespace=~".+"}[1m]))+ sum(rate(container_fs_writes_total{pod=~"openshift-gitops-application-controller-.*",namespace=~".+"}[1m]))
+  metricName: argocdAppControllerIO
+
+
+#  ArgoCD Pending Kubectl Exec
+- query: sum(argocd_kubectl_exec_pending{namespace=~".+"})
+  metricName: argocdPendingKubectlExec
+
+
+
+

tools/kube-burner/readme.md

@@ -0,0 +1,167 @@
+# ArgoCD Performance Testing with Kube-Burner
+
+This repository contains the configuration that is used to test ArgoCD's performance when managing multiple applications across different namespaces.
+
+## Prerequisites
+
+- **ArgoCD Installation**: Ensure that ArgoCD is installed and configured in your cluster.
+- **OpenShift Cluster**: This setup is designed for an OpenShift cluster, but it should work on any Kubernetes-compatible environment.
+- **Kube-Burner**: Install Kube-Burner for resource creation and metrics collection. You can find installation instructions on the [Kube-Burner GitHub page](https://github.com/kube-burner/kube-burner).
+
+## Why Use Kube-Burner?
+
+[Kube-Burner](https://kube-burner.github.io/kube-burner/latest/) is a versatile tool that creates resources, waits for them to reach a desired state, and collects relevant metrics. This makes it ideal for performance testing and benchmarking. In the context of ArgoCD, Kube-Burner can efficiently manage the lifecycle of ArgoCD applications by creating them based on predefined templates, waiting for their status to meet specific conditions (such as synchronization), and gathering key metrics. For example, the configuration below specifies the creation of ArgoCD Application resources, waits until they are in a "Synced" state using the customStatusPath for detailed status checks, and collects relevant metrics for performance analysis:
+
+### Wait Configuration Example
+
+The following example illustrates the wait configuration used in this setup:
+
+```yaml
+objects:
+      - kind: Application
+        objectTemplate: template/application.yaml
+        replicas: 1
+        waitOptions:
+          forCondition: "Synced"
+          customStatusPath: ".operationState.syncResult.resources[].status"
+ ```
+
+### Kube-Burner Configuration Details
+
+#### Metrics Profile
+
+The metric/metrics-profile.yaml file defines the metrics profile for ArgoCD performance testing, capturing essential metrics:
+
+- ***CPU Usage of ArgoCD Application Controller:***
+```
+- query: irate(process_cpu_seconds_total{container="argocd-application-controller",namespace=~".+"}[1m])
+  metricName: argocdAppControllerCPU
+```
+- ***Memory Usage of ArgoCD Application Controller:***
+```
+- query: go_memstats_heap_alloc_bytes{container="argocd-application-controller",namespace=~".+"}
+  metricName: argocdAppControllerHeapAllocMemory
+
+- query: go_memstats_heap_inuse_bytes{container="argocd-application-controller",namespace=~".+"}
+  metricName: argocdAppControllerHeapInuseMemory
+```
+- ***Number of Kubernetes Resource Objects in the Cache:***
+```
+- query: sum(argocd_cluster_api_resource_objects{namespace=~".+"})
+  metricName: argocdClusterApiResourceObjects
+```  
+- ***Number of Monitored Kubernetes API Resources:***
+```
+- query: sum(argocd_cluster_api_resources{namespace=~".+"})
+  metricName: argocdClusterApiResources
+```
+- ***Total IO Operations of ArgoCD Application Controller:***
+```
+- query: sum(rate(container_fs_reads_total{pod=~"openshift-gitops-application-controller-.*",namespace=~".+"}[1m])) + sum(rate(container_fs_writes_total{pod=~"openshift-gitops-application-controller-.*",namespace=~".+"}[1m]))
+  metricName: argocdAppControllerIO
+```
+- ***ArgoCD Pending Kubectl Exec:***
+```
+- query: sum(argocd_kubectl_exec_pending{namespace=~".+"})
+  metricName: argocdPendingKubectlExec
+```
+
+### Sample Metrics Data
+
+Below is a sample of metrics data collected using Kube-Burner:
+
+```json
+[
+    {
+        "timestamp": "2024-08-22T10:17:39.867Z",
+        "labels": {
+            "container": "argocd-application-controller",
+            "endpoint": "metrics",
+            "instance": "10.131.1.218:8082",
+            "job": "openshift-gitops-metrics",
+            "namespace": "openshift-gitops",
+            "pod": "openshift-gitops-application-controller-0",
+            "service": "openshift-gitops-metrics"
+        },
+        "value": 1281433176,
+        "uuid": "1234",
+        "query": "go_memstats_heap_alloc_bytes{container=\"argocd-application-controller\",namespace=~\".+\"}",
+        "metricName": "argocdAppControllerHeapAllocMemory",
+        "jobName": "create-applications"
+    },
+    {
+        "timestamp": "2024-08-22T10:16:53.241Z",
+        "labels": {
+            "container": "argocd-application-controller",
+            "endpoint": "metrics",
+            "instance": "10.131.1.218:8082",
+            "job": "openshift-gitops-metrics",
+            "namespace": "openshift-gitops",
+            "pod": "openshift-gitops-application-controller-0",
+            "service": "openshift-gitops-metrics"
+        },
+        "value": 188994960,
+        "uuid": "1234",
+        "query": "go_memstats_heap_alloc_bytes{container=\"argocd-application-controller\",namespace=~\".+\"}",
+        "metricName": "argocdAppControllerHeapAllocMemory",
+        "jobName": "create-namespaces"
+    },
+    {
+        "timestamp": "2024-08-22T10:17:04.303Z",
+        "labels": {
+            "container": "argocd-application-controller",
+            "endpoint": "metrics",
+            "instance": "10.131.1.218:8082",
+            "job": "openshift-gitops-metrics",
+            "namespace": "openshift-gitops",
+            "pod": "openshift-gitops-application-controller-0",
+            "service": "openshift-gitops-metrics"
+        },
+        "value": 202814368,
+        "uuid": "1234",
+        "query": "go_memstats_heap_alloc_bytes{container=\"argocd-application-controller\",namespace=~\".+\"}",
+        "metricName": "argocdAppControllerHeapAllocMemory",
+        "jobName": "create-role"
+    }
+]
+```
+
+### Metric Endpoint Configuration
+
+To configure metric collection, update the metric/ep.yaml file as follows:
+```yaml
+endpoint: <metric-endpoint>
+token: ""
+metrics:
+  - metric/metrics-profile.yaml # Path to the metrics query profile
+indexer:
+  type: local
+  metricsDirectory: my-metrics # Directory where metrics will be collected
+``` 
+Replace <metric-endpoint> with the appropriate endpoint for your setup.
+
+### Template Customization
+
+Customize the YAML templates in the template directory before deployment:
+
+- application.yaml: Update the namespace field with ```<argocd-namespace>```.
+
+- namespace.yaml:
+```yaml
+labels:
+  argocd.argoproj.io/managed-by: <argocd-namespace>
+```  
+- rolebinding.yaml:
+```yaml
+subjects:
+  - kind: ServiceAccount
+    name: <application-controller-service-account-name>
+    namespace: <argocd-namespace>
+```
+
+Replace ```<argocd-namespace>``` and ```<application-controller-service-account-name>``` with the values relevant to your environment.
+
+### Deploying the Test Setup
+To deploy and start the performance test, navigate to the kube-burner directory and execute:
+
+``` kube-burner init -c config.yaml -e metric/ep.yaml ```

tools/kube-burner/sample-test-resource/deployment.yaml

@@ -0,0 +1,25 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    kube-burner: argocd-performance-test
+    app: bubble-animation
+  name: bubble-animation
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: bubble-animation
+  strategy: {}
+  template:
+    metadata:
+      labels:
+        app: bubble-animation
+    spec:
+      containers:
+      - image: quay.io/rhdevelopers/bgd:1.0.0
+        name: bubble-animation
+        env:
+        - name: COLOR
+          value: "blue"

tools/kube-burner/sample-test-resource/service.yaml

@@ -0,0 +1,14 @@
+---
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app: bubble-animation
+  name: bubble-animation
+spec:
+  ports:
+  - port: 8081
+    protocol: TCP
+    targetPort: 8080
+  selector:
+    app: bubble-animation

tools/kube-burner/template/application.yaml

@@ -0,0 +1,16 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: app-{{.Iteration}}
+  namespace: <argocd-namespace>
+spec:
+  project: default
+  source:
+    repoURL: https://github.com/argoproj-labs/argocd-performance-test
+    targetRevision: HEAD
+    path: tools/kube-burner/sample-test-resource
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: argocd-test-{{.Iteration}}
+  syncPolicy:
+    automated: {}

tools/kube-burner/template/namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  labels:
+    argocd.argoproj.io/managed-by: <argocd-namespace>
+  name: argocd-test-{{.Iteration}}

tools/kube-burner/template/role.yaml

@@ -0,0 +1,13 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: argocd-role
+  namespace: argocd-test-{{.Iteration}}
+rules:
+  - apiGroups: [""]
+    resources: ["services"]
+    verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
+

tools/kube-burner/template/rolebinding.yaml

@@ -0,0 +1,13 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: argocd-rolebinding
+  namespace: argocd-test-{{.Iteration}}
+subjects:
+  - kind: ServiceAccount
+    name: <application-controller-service-account-name>
+    namespace: <argocd-namespace>
+roleRef:
+  kind: Role
+  name: argocd-role
+  apiGroup: rbac.authorization.k8s.io