diff --git a/README.md b/README.md index 3ad2d14..56a2b5c 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ ## Current Tools -### ArgoCD/ArgoWorkflows tooling based on Amazon EKS - [Link](tools/awsbenchmarking-nodejs) +### 1. ArgoCD/ArgoWorkflows tooling based on Amazon EKS - [Link](tools/awsbenchmarking-nodejs) This tooling is built on nodejs and utilizes various AWS tools to deploy and run benchmarks for both ArgoCD and ArgoWorkflows on Amazon EKS. This is the same tooling utilized to capture data for the following blog posts/talks: * [Argo CD Application Controller Scalability Testing on Amazon EKS](https://aws.amazon.com/blogs/opensource/argo-cd-application-controller-scalability-testing-on-amazon-eks/) @@ -10,4 +10,8 @@ This tooling is built on nodejs and utilizes various AWS tools to deploy and run * [Argo Workflows Controller Scalability Testing on Amazon EKS](https://cnoe.io/blog/argo-workflow-scalability) * [Adobe/AWS: Key Takeaways from Scaling Adobe's CI/CD Solution to Support 50K Argo CD Apps ](https://www.youtube.com/watch?v=7yVXMCX62tY) +### 2. Kube-Burner - [Link](tools/kube-burner) + +[Kube-Burner](https://kube-burner.github.io/kube-burner/latest/) is a resource creation and metrics collection tool. It is used to test ArgoCD performance by creating multiple ArgoCD Applications and collecting performance metrics such as CPU, memory, and API server requests. For detailed setup and usage, check the [Kube-Burner README](tools/kube-burner/readme.md). + diff --git a/tools/kube-burner/config.yaml b/tools/kube-burner/config.yaml new file mode 100644 index 0000000..ae2b2b9 --- /dev/null +++ b/tools/kube-burner/config.yaml @@ -0,0 +1,48 @@ +# Global configuration +global: + gc: false + +# Jobs configuration +jobs: + # Job to create Namespaces, Roles, and RoleBindings + - name: create-namespaces + jobIterations: 100 + cleanup: false + qps: 10 # Increased qps to handle more requests + burst: 50 # Increased burst to allow higher initial throughput + objects: + - kind: Namespace + objectTemplate: template/namespace.yaml + replicas: 1 + - name: create-role + jobIterations: 100 + cleanup: false + qps: 10 + burst: 50 + objects: + - kind: Role + objectTemplate: template/role.yaml + replicas: 1 + - name: create-rolebinding + jobIterations: 100 + cleanup: false + qps: 10 + burst: 50 + objects: + - kind: RoleBinding + objectTemplate: template/rolebinding.yaml + replicas: 1 + # Job to create ArgoCD applications + - name: create-applications + jobIterations: 100 + cleanup: false + waitWhenFinished: true + qps: 10 + burst: 50 + objects: + - kind: Application + objectTemplate: template/application.yaml + replicas: 1 + waitOptions: + forCondition: "Synced" + customStatusPath: ".operationState.syncResult.resources[].status" diff --git a/tools/kube-burner/metric/ep.yaml b/tools/kube-burner/metric/ep.yaml new file mode 100644 index 0000000..4f04467 --- /dev/null +++ b/tools/kube-burner/metric/ep.yaml @@ -0,0 +1,7 @@ +- endpoint: + token: "" + metrics: + - metric/metrics-profile.yaml + indexer: + type: local + metricsDirectory: my-metrics \ No newline at end of file diff --git a/tools/kube-burner/metric/metrics-profile.yaml b/tools/kube-burner/metric/metrics-profile.yaml new file mode 100644 index 0000000..62d5e42 --- /dev/null +++ b/tools/kube-burner/metric/metrics-profile.yaml @@ -0,0 +1,34 @@ +# Metrics Profile for ArgoCD Performance Testing + +# CPU Usage of ArgoCD Application Controller +- query: irate(process_cpu_seconds_total{container="argocd-application-controller",namespace=~".+"}[1m]) + metricName: argocdAppControllerCPU + +# Memory Usage of ArgoCD Application Controller +- query: go_memstats_heap_alloc_bytes{container="argocd-application-controller",namespace=~".+"} + metricName: argocdAppControllerHeapAllocMemory + +- query: go_memstats_heap_inuse_bytes{container="argocd-application-controller",namespace=~".+"} + metricName: argocdAppControllerHeapInuseMemory + + +# Number of k8s resource objects in the cache +- query: sum(argocd_cluster_api_resource_objects{namespace=~".+"}) + metricName: argocdClusterApiResourceObjects + +# Number of monitored Kubernetes API resources +- query: sum(argocd_cluster_api_resources{namespace=~".+"}) + metricName: argocdClusterApiResources + +# Total IO Operations of ArgoCD Application Controller +- query: sum(rate(container_fs_reads_total{pod=~"openshift-gitops-application-controller-.*",namespace=~".+"}[1m]))+ sum(rate(container_fs_writes_total{pod=~"openshift-gitops-application-controller-.*",namespace=~".+"}[1m])) + metricName: argocdAppControllerIO + + +# ArgoCD Pending Kubectl Exec +- query: sum(argocd_kubectl_exec_pending{namespace=~".+"}) + metricName: argocdPendingKubectlExec + + + + diff --git a/tools/kube-burner/readme.md b/tools/kube-burner/readme.md new file mode 100644 index 0000000..0374fc7 --- /dev/null +++ b/tools/kube-burner/readme.md @@ -0,0 +1,167 @@ +# ArgoCD Performance Testing with Kube-Burner + +This repository contains the configuration that is used to test ArgoCD's performance when managing multiple applications across different namespaces. + +## Prerequisites + +- **ArgoCD Installation**: Ensure that ArgoCD is installed and configured in your cluster. +- **OpenShift Cluster**: This setup is designed for an OpenShift cluster, but it should work on any Kubernetes-compatible environment. +- **Kube-Burner**: Install Kube-Burner for resource creation and metrics collection. You can find installation instructions on the [Kube-Burner GitHub page](https://github.com/kube-burner/kube-burner). + +## Why Use Kube-Burner? + +[Kube-Burner](https://kube-burner.github.io/kube-burner/latest/) is a versatile tool that creates resources, waits for them to reach a desired state, and collects relevant metrics. This makes it ideal for performance testing and benchmarking. In the context of ArgoCD, Kube-Burner can efficiently manage the lifecycle of ArgoCD applications by creating them based on predefined templates, waiting for their status to meet specific conditions (such as synchronization), and gathering key metrics. For example, the configuration below specifies the creation of ArgoCD Application resources, waits until they are in a "Synced" state using the customStatusPath for detailed status checks, and collects relevant metrics for performance analysis: + +### Wait Configuration Example + +The following example illustrates the wait configuration used in this setup: + +```yaml +objects: + - kind: Application + objectTemplate: template/application.yaml + replicas: 1 + waitOptions: + forCondition: "Synced" + customStatusPath: ".operationState.syncResult.resources[].status" + ``` + +### Kube-Burner Configuration Details + +#### Metrics Profile + +The metric/metrics-profile.yaml file defines the metrics profile for ArgoCD performance testing, capturing essential metrics: + +- ***CPU Usage of ArgoCD Application Controller:*** +``` +- query: irate(process_cpu_seconds_total{container="argocd-application-controller",namespace=~".+"}[1m]) + metricName: argocdAppControllerCPU +``` +- ***Memory Usage of ArgoCD Application Controller:*** +``` +- query: go_memstats_heap_alloc_bytes{container="argocd-application-controller",namespace=~".+"} + metricName: argocdAppControllerHeapAllocMemory + +- query: go_memstats_heap_inuse_bytes{container="argocd-application-controller",namespace=~".+"} + metricName: argocdAppControllerHeapInuseMemory +``` +- ***Number of Kubernetes Resource Objects in the Cache:*** +``` +- query: sum(argocd_cluster_api_resource_objects{namespace=~".+"}) + metricName: argocdClusterApiResourceObjects +``` +- ***Number of Monitored Kubernetes API Resources:*** +``` +- query: sum(argocd_cluster_api_resources{namespace=~".+"}) + metricName: argocdClusterApiResources +``` +- ***Total IO Operations of ArgoCD Application Controller:*** +``` +- query: sum(rate(container_fs_reads_total{pod=~"openshift-gitops-application-controller-.*",namespace=~".+"}[1m])) + sum(rate(container_fs_writes_total{pod=~"openshift-gitops-application-controller-.*",namespace=~".+"}[1m])) + metricName: argocdAppControllerIO +``` +- ***ArgoCD Pending Kubectl Exec:*** +``` +- query: sum(argocd_kubectl_exec_pending{namespace=~".+"}) + metricName: argocdPendingKubectlExec +``` + +### Sample Metrics Data + +Below is a sample of metrics data collected using Kube-Burner: + +```json +[ + { + "timestamp": "2024-08-22T10:17:39.867Z", + "labels": { + "container": "argocd-application-controller", + "endpoint": "metrics", + "instance": "10.131.1.218:8082", + "job": "openshift-gitops-metrics", + "namespace": "openshift-gitops", + "pod": "openshift-gitops-application-controller-0", + "service": "openshift-gitops-metrics" + }, + "value": 1281433176, + "uuid": "1234", + "query": "go_memstats_heap_alloc_bytes{container=\"argocd-application-controller\",namespace=~\".+\"}", + "metricName": "argocdAppControllerHeapAllocMemory", + "jobName": "create-applications" + }, + { + "timestamp": "2024-08-22T10:16:53.241Z", + "labels": { + "container": "argocd-application-controller", + "endpoint": "metrics", + "instance": "10.131.1.218:8082", + "job": "openshift-gitops-metrics", + "namespace": "openshift-gitops", + "pod": "openshift-gitops-application-controller-0", + "service": "openshift-gitops-metrics" + }, + "value": 188994960, + "uuid": "1234", + "query": "go_memstats_heap_alloc_bytes{container=\"argocd-application-controller\",namespace=~\".+\"}", + "metricName": "argocdAppControllerHeapAllocMemory", + "jobName": "create-namespaces" + }, + { + "timestamp": "2024-08-22T10:17:04.303Z", + "labels": { + "container": "argocd-application-controller", + "endpoint": "metrics", + "instance": "10.131.1.218:8082", + "job": "openshift-gitops-metrics", + "namespace": "openshift-gitops", + "pod": "openshift-gitops-application-controller-0", + "service": "openshift-gitops-metrics" + }, + "value": 202814368, + "uuid": "1234", + "query": "go_memstats_heap_alloc_bytes{container=\"argocd-application-controller\",namespace=~\".+\"}", + "metricName": "argocdAppControllerHeapAllocMemory", + "jobName": "create-role" + } +] +``` + +### Metric Endpoint Configuration + +To configure metric collection, update the metric/ep.yaml file as follows: +```yaml +endpoint: +token: "" +metrics: + - metric/metrics-profile.yaml # Path to the metrics query profile +indexer: + type: local + metricsDirectory: my-metrics # Directory where metrics will be collected +``` +Replace with the appropriate endpoint for your setup. + +### Template Customization + +Customize the YAML templates in the template directory before deployment: + +- application.yaml: Update the namespace field with ``````. + +- namespace.yaml: +```yaml +labels: + argocd.argoproj.io/managed-by: +``` +- rolebinding.yaml: +```yaml +subjects: + - kind: ServiceAccount + name: + namespace: +``` + +Replace `````` and `````` with the values relevant to your environment. + +### Deploying the Test Setup +To deploy and start the performance test, navigate to the kube-burner directory and execute: + +``` kube-burner init -c config.yaml -e metric/ep.yaml ``` diff --git a/tools/kube-burner/sample-test-resource/deployment.yaml b/tools/kube-burner/sample-test-resource/deployment.yaml new file mode 100644 index 0000000..bb52860 --- /dev/null +++ b/tools/kube-burner/sample-test-resource/deployment.yaml @@ -0,0 +1,25 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + kube-burner: argocd-performance-test + app: bubble-animation + name: bubble-animation +spec: + replicas: 1 + selector: + matchLabels: + app: bubble-animation + strategy: {} + template: + metadata: + labels: + app: bubble-animation + spec: + containers: + - image: quay.io/rhdevelopers/bgd:1.0.0 + name: bubble-animation + env: + - name: COLOR + value: "blue" diff --git a/tools/kube-burner/sample-test-resource/service.yaml b/tools/kube-burner/sample-test-resource/service.yaml new file mode 100644 index 0000000..3f38923 --- /dev/null +++ b/tools/kube-burner/sample-test-resource/service.yaml @@ -0,0 +1,14 @@ +--- +apiVersion: v1 +kind: Service +metadata: + labels: + app: bubble-animation + name: bubble-animation +spec: + ports: + - port: 8081 + protocol: TCP + targetPort: 8080 + selector: + app: bubble-animation diff --git a/tools/kube-burner/template/application.yaml b/tools/kube-burner/template/application.yaml new file mode 100644 index 0000000..7270628 --- /dev/null +++ b/tools/kube-burner/template/application.yaml @@ -0,0 +1,16 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: app-{{.Iteration}} + namespace: +spec: + project: default + source: + repoURL: https://github.com/argoproj-labs/argocd-performance-test + targetRevision: HEAD + path: tools/kube-burner/sample-test-resource + destination: + server: https://kubernetes.default.svc + namespace: argocd-test-{{.Iteration}} + syncPolicy: + automated: {} diff --git a/tools/kube-burner/template/namespace.yaml b/tools/kube-burner/template/namespace.yaml new file mode 100644 index 0000000..ac1342a --- /dev/null +++ b/tools/kube-burner/template/namespace.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + labels: + argocd.argoproj.io/managed-by: + name: argocd-test-{{.Iteration}} diff --git a/tools/kube-burner/template/role.yaml b/tools/kube-burner/template/role.yaml new file mode 100644 index 0000000..73cfa3c --- /dev/null +++ b/tools/kube-burner/template/role.yaml @@ -0,0 +1,13 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: argocd-role + namespace: argocd-test-{{.Iteration}} +rules: + - apiGroups: [""] + resources: ["services"] + verbs: ["create", "get", "list", "watch", "update", "patch", "delete"] + - apiGroups: ["apps"] + resources: ["deployments"] + verbs: ["create", "get", "list", "watch", "update", "patch", "delete"] + diff --git a/tools/kube-burner/template/rolebinding.yaml b/tools/kube-burner/template/rolebinding.yaml new file mode 100644 index 0000000..f9a2939 --- /dev/null +++ b/tools/kube-burner/template/rolebinding.yaml @@ -0,0 +1,13 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: argocd-rolebinding + namespace: argocd-test-{{.Iteration}} +subjects: + - kind: ServiceAccount + name: + namespace: +roleRef: + kind: Role + name: argocd-role + apiGroup: rbac.authorization.k8s.io