Add test horizontal-pod-autoscaling

.gitignore

@@ -1,7 +1,7 @@
 .env
 *.npz
 index.faiss
-test*
+# test*
 # *.ipynb
 NOTE.md
 elastic_search_data

backend/locust/locustfile.py

@@ -0,0 +1,25 @@
+from typing import Any
+
+from locust import FastHttpUser, task
+
+
+class SearchImageUser(FastHttpUser):
+    host = "http://search.vectornguyen.com/backend"
+
+    def __init__(self, *args: Any, **kwargs: Any):
+        super().__init__(*args, **kwargs)
+
+    @task
+    def healthz(self):
+        self.client.get("/healthz")
+
+    @task
+    def create_user(self):
+        data = {
+            "email": "phuoc@gmail.com",
+            "username": "Vector Nguyen",
+            "image": "",
+            "token": "",
+        }
+
+        self.client.post("/auth/users/tokens", json=data)

helm-charts/README.md

@@ -8,7 +8,8 @@ Helm is a package manager for Kubernetes, simplifying the process of defining, i
 2. [Create Cluster and NodeGroup](#create-cluster-and-nodegroup)
 3. [Model Repository S3](#model-repository-s3)
 4. [Install aws-ebs-csi-driver](#install-aws-ebs-csi-driver)
-5. [Install Charts](#install-charts)
+5. [Install Metric Server](#install-metric-server)
+6. [Install Charts](#install-charts)
    - [Ingress Nginx Controller](#ingress-nginx-controller)
    - [Postgresql](#postgresql)
    - [Elastic Search](#elastic-search)
@@ -19,11 +20,13 @@ Helm is a package manager for Kubernetes, simplifying the process of defining, i
    - [Text Search Application](#text-search-application)
    - [Backend Application](#backend-application)
    - [Frontend Application](#frontend-application)
-6. [Upgrade Charts](#upgrade-charts)
-7. [Uninstall Charts](#uninstall-charts)
-8. [Clean up PVC](#clean-up-pvc)
-9. [Check Resources](#check-resources)
-10. [References](#references)
+7. [Load Test Horizontal Pod Autoscaling](#load-test-horizontal-pod-autoscaling)
+   - [Test Backend Horizontal Pod Autoscaling](#test-backend-horizontal-pod-autoscaling)
+8. [Upgrade Charts](#upgrade-charts)
+9. [Uninstall Charts](#uninstall-charts)
+10. [Clean up PVC](#clean-up-pvc)
+11. [Check Resources](#check-resources)
+12. [References](#references)
 
 ## Architecture
 
@@ -95,6 +98,32 @@ kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernete
 
    Review the [configuration values](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/charts/aws-ebs-csi-driver/values.yaml) for the Helm chart.
 
+## Install Metric Server
+Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through [Metrics API]
+for use by [Horizontal Pod Autoscaler] and [Vertical Pod Autoscaler]. Metrics API can also be accessed by `kubectl top`,
+making it easier to debug autoscaling pipelines.
+
+[Metrics API]: https://github.com/kubernetes/metrics
+[Horizontal Pod Autoscaler]: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
+[Vertical Pod Autoscaler]: https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/
+
+1. **Kustomize**
+```sh
+kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
+```
+
+2. **Helm**
+- Add the `metrics-server` Helm repository.
+   ```sh
+   helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
+
+   ```
+
+- Install the latest release.
+   ```sh
+   helm upgrade --install metrics-server metrics-server/metrics-server
+   ```
+
 ## Install Charts
 Step-by-step instructions to create namespaces and install various Helm charts like Ingress Nginx Controller, Postgresql, Elastic Search, Qdrant, Prometheus, Grafana, and others.
 
@@ -186,6 +215,65 @@ Step-by-step instructions to create namespaces and install various Helm charts l
    helm install frontend-app ./frontend --namespace application --set nodeSelector.nodegroup-type=cpu-nodegroup
    ```
 
+## Load Test Horizontal Pod Autoscaling
+
+### Test Backend Horizontal Pod Autoscaling
+
+1. **Use Locust for Load Test**
+
+   Navigate to the `locust` directory within the backend application and run Locust.
+
+   ```bash
+   cd ../backend/locust
+   locust
+   ```
+
+2. **Access Locust Web Interface**
+   Visit `http://localhost:8089` in your web browser to access the Locust web interface.
+
+   <p align="center">
+   <img src="./assets/start-locust.png" alt="Start Locust" />
+   <br>
+   <em>Fig: Start Locust</em>
+   </p>
+
+3. **Track Backend Application Scaling**
+   Run the following command to monitor the Horizontal Pod Autoscaler (HPA) for the backend application.
+
+   ```bash
+   kubectl get hpa backend-app --namespace application --watch
+   ```
+
+   - **Scale Up on Increased Load:**
+
+      As the number of users increases in the Locust test, observe the backend-app pod scaling up.
+
+      <p align="center">
+      <img src="./assets/test-hpa-scale-up.png" alt="Test HPA Scale Up" />
+      <br>
+      <em>Fig: Test HPA Scale Up</em>
+      </p>
+
+   - **Scale Down after Load Stops:**
+
+      When the Locust test is stopped, monitor the backend-app pod scaling down.
+
+      <p align="center">
+      <img src="./assets/test-hpa-scale-down.png" alt="Test HPA Scale Down" />
+      <br>
+      <em>Fig: Test HPA Scale Down</em>
+      </p>
+
+   - **Locust Test in Progress:**
+
+      View the Locust test results on the web interface.
+
+      <p align="center">
+      <img src="./assets/locust-test.png" alt="Locust Test" />
+      <br>
+      <em>Fig: Locust Test</em>
+      </p>
+
 ## Upgrade Charts
 Instructions on how to upgrade existing Helm Chart releases.
 
@@ -276,4 +364,5 @@ Deleting PVCs is irreversible and can lead to data loss. Ensure backups are in p
 - [Prometheus Community Helm Charts](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)
 - [Qdrant Helm Charts](https://github.com/qdrant/qdrant-helm/tree/main/charts/qdrant)
 - [Elastic Cloud on Kubernetes Documentation](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-stack-helm-chart.html)
-- [CSI driver for Amazon EBS](https://github.com/kubernetes-sigs/aws-ebs-csi-driver)
+- [CSI driver for Amazon EBS](https://github.com/kubernetes-sigs/aws-ebs-csi-driver)
+- [Kubernetes Metrics Server](https://github.com/kubernetes-sigs/metrics-server)

helm-charts/backend/templates/deployment.yaml

@@ -57,6 +57,9 @@ spec:
               port: http
             initialDelaySeconds: 5
             periodSeconds: 5
+          # Only limit resources for test hpa
+          resources:
+            {{- toYaml .Values.resources | nindent 12 }}
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}

helm-charts/backend/templates/tests/test-connection.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: "{{ include "backend.fullname" . }}-test-connection"
+  labels:
+    {{- include "backend.labels" . | nindent 4 }}
+  annotations:
+    "helm.sh/hook": test
+spec:
+  containers:
+    - name: wget
+      image: busybox
+      command: ['wget']
+      args: ['{{ include "backend.fullname" . }}:{{ .Values.service.port }}']
+  restartPolicy: Never

helm-charts/backend/values.yaml

@@ -36,6 +36,16 @@ env:
       secretKeyRef:
         key: JWT_SECRET
 
+
+# Only limit resources for test hpa
+resources: 
+  limits:
+    cpu: 500m
+    memory: 256Mi
+  requests:
+    cpu: 500m
+    memory: 256Mi
+
 secret:
   userName: db_user
   postgresPassword: db_password

helm-charts/frontend/templates/tests/test-connection.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: "{{ include "frontend.fullname" . }}-test-connection"
+  labels:
+    {{- include "frontend.labels" . | nindent 4 }}
+  annotations:
+    "helm.sh/hook": test
+spec:
+  containers:
+    - name: wget
+      image: busybox
+      command: ['wget']
+      args: ['{{ include "frontend.fullname" . }}:{{ .Values.service.port }}']
+  restartPolicy: Never

helm-charts/image-search/templates/tests/test-connection.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: "{{ include "image-search.fullname" . }}-test-connection"
+  labels:
+    {{- include "image-search.labels" . | nindent 4 }}
+  annotations:
+    "helm.sh/hook": test
+spec:
+  containers:
+    - name: wget
+      image: busybox
+      command: ['wget']
+      args: ['{{ include "image-search.fullname" . }}:{{ .Values.service.port }}']
+  restartPolicy: Never

helm-charts/qdrant/templates/tests/test-db-interaction.yaml

@@ -0,0 +1,98 @@
+{{- $root := . }}
+{{- $namespace := .Release.Namespace }}
+apiVersion: v1
+kind: Pod
+metadata:
+  name: "{{ include "qdrant.fullname" . }}-test-db-interaction"
+  labels:
+    {{- include "qdrant.labels" . | nindent 4 }}
+  annotations:
+    "helm.sh/hook": test
+spec:
+  containers:
+    - name: test-script
+      image: registry.suse.com/bci/bci-base:latest
+      args: ['bash', '/app/entrypoint.sh']
+      volumeMounts:
+        - mountPath: /app
+          name: test-script
+        {{- if .Values.additionalVolumeMounts }}
+{{- toYaml .Values.additionalVolumeMounts | default "" | nindent 8 }}
+        {{- end}}
+  volumes:
+    - name: test-script
+      configMap:
+        name: "{{ include "qdrant.fullname" . }}-test-db-interaction"
+    {{- if .Values.additionalVolumes }}
+{{- toYaml .Values.additionalVolumes | default "" | nindent 4 }}
+    {{- end}}
+  restartPolicy: Never
+  serviceAccountName: {{ include "qdrant.fullname" . }}
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: "{{ include "qdrant.fullname" . }}-test-db-interaction"
+  labels:
+    {{- include "qdrant.labels" . | nindent 4 }}
+  annotations:
+    "helm.sh/hook": test
+data:
+  entrypoint.sh: |
+    #!/bin/bash
+    set -xe
+    echo 'connect-timeout = 5' > $HOME/.curlrc
+    echo 'retry = 5' >> $HOME/.curlrc
+    echo 'retry-all-errors' >> $HOME/.curlrc
+
+    if [ -d /mnt/secrets/certs ]; then
+      cp /mnt/secrets/certs/ca.pem /usr/share/pki/trust/anchors/private-ca.pem
+      update-ca-certificates
+    fi
+
+    QDRANT_COLLECTION="test_collection"
+    {{- range .Values.service.ports }}
+    {{- if eq .name "http" }}
+    echo "Connecting to {{ include "qdrant.fullname" $root }}.{{ $namespace }}:{{ .port }}"
+    QDRANT_URL="http://{{ include "qdrant.fullname" $root }}.{{ $namespace }}:{{ .port }}"
+    {{- if and $root.Values.config.service $root.Values.config.service.enable_tls }}
+    echo "Using https"
+    QDRANT_URL="https://{{ include "qdrant.fullname" $root }}.{{ $namespace }}:{{ .port }}"
+    {{- end }}
+    {{- end }}
+    {{- end }}
+    API_KEY_HEADER=""
+    {{- if .Values.apiKey }}
+    API_KEY_HEADER="Api-key: {{ .Values.apiKey }}"
+    {{- end }}
+
+    # Delete collection if exists
+    curl -X DELETE -H "${API_KEY_HEADER}" $QDRANT_URL/collections/${QDRANT_COLLECTION}
+
+    # Create collection
+    curl -X PUT \
+    -H 'Content-Type: application-json' \
+    -d '{"vectors":{"size":4,"distance":"Dot"}}' \
+    -H  "${API_KEY_HEADER}" \
+    $QDRANT_URL/collections/${QDRANT_COLLECTION} --fail-with-body
+
+    # Insert points
+    curl -X PUT \
+    -H 'Content-Type: application-json' \
+    -d '{"points":[
+      {"id":1,"vector":[0.05, 0.61, 0.76, 0.74],"payload":{"city":"Berlin"}},
+      {"id":2,"vector":[0.19, 0.81, 0.75, 0.11],"payload":{"city":"London"}},
+      {"id":3,"vector":[0.36, 0.55, 0.47, 0.94],"payload":{"city":"Moscow"}},
+      {"id":4,"vector":[0.18, 0.01, 0.85, 0.80],"payload":{"city":"New York"}},
+      {"id":5,"vector":[0.24, 0.18, 0.22, 0.44],"payload":{"city":"Beijing"}},
+      {"id":6,"vector":[0.35, 0.08, 0.11, 0.44],"payload":{"city":"Mumbai"}}
+    ]}' \
+    -H  "${API_KEY_HEADER}" \
+    $QDRANT_URL/collections/${QDRANT_COLLECTION}/points --fail-with-body
+
+    # Run query
+    curl -X POST \
+    -H 'Content-Type: application-json' \
+    -d '{"vector":[0.2, 0.1, 0.9, 0.7],"limit":3}' \
+    -H  "${API_KEY_HEADER}" \
+    $QDRANT_URL/collections/${QDRANT_COLLECTION}/points/search --fail-with-body

helm-charts/text-search/templates/tests/test-connection.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: "{{ include "text-search.fullname" . }}-test-connection"
+  labels:
+    {{- include "text-search.labels" . | nindent 4 }}
+  annotations:
+    "helm.sh/hook": test
+spec:
+  containers:
+    - name: wget
+      image: busybox
+      command: ['wget']
+      args: ['{{ include "text-search.fullname" . }}:{{ .Values.service.port }}']
+  restartPolicy: Never