Merge pull request #321 from akgalwas/add-scripts-for-shoot-comparison-2

Add job that executes shoot comparison

hack/shoot-comparator/scripts/README.md

@@ -0,0 +1,126 @@
+# Shoot comparator scripts
+
+## Overview
+The scripts are designed to make shoot comparison easier. 
+
+## Running comparison scripts
+
+In order to run the comparison scripts, execute the following command:
+```bash
+./run-comparison.sh
+```
+
+The script will perform the following steps:
+- Invoke ./cleanup-comparison-resources.sh to clean up the resources created during the previous comparison
+- Prepare volume snapshots and PVCs containing files stored by Provisioner and KIM
+- Run the comparison job that mounts the PVCs and compares the files. Results will be stored in the separate volume.
+- Prepare volume snapshot and PVC containing the comparison results
+- Run pod that mounts the PVC
+- Copy the comparison results to the local directory
+
+The following is an example of the script output:
+```
+Cleaning up the resources created for shoot comparison
+
+Removing resources needed for fetching results
+pod "fetch-test-comparison-results" deleted
+persistentvolumeclaim "shoot-comparator-pvc-read-only" deleted
+volumesnapshot.snapshot.storage.k8s.io "shoot-comparator-pvc" deleted
+
+Removing resources needed for performing comparison
+job.batch "compare-shoots" deleted
+persistentvolumeclaim "shoot-comparator-pvc" deleted
+persistentvolumeclaim "test-prov-shoot-read-only" deleted
+persistentvolumeclaim "test-kim-shoot-read-only" deleted
+volumesnapshot.snapshot.storage.k8s.io "test-kim-shoot-spec-storage" deleted
+volumesnapshot.snapshot.storage.k8s.io "test-prov-shoot-spec-storage" deleted
+
+Preparing data for comparison
+volumesnapshot.snapshot.storage.k8s.io/test-prov-shoot-spec-storage created
+volumesnapshot.snapshot.storage.k8s.io/test-kim-shoot-spec-storage created
+persistentvolumeclaim/test-prov-shoot-read-only created
+persistentvolumeclaim/test-kim-shoot-read-only created
+
+Running comparison job
+persistentvolumeclaim/shoot-comparator-pvc created
+job.batch/compare-shoots created
+
+Waiting for the job to complete. It may take couple of minutes. Please, be patient!
+job.batch/compare-shoots condition met
+Job completed
+
+Fetching logs for the job
+2024/08/01 11:10:56 INFO Comparing directories: /testdata/provisioner and /testdata/kim
+2024/08/01 11:10:56 INFO Saving comparison details
+2024/08/01 11:10:56 INFO Results stored in "/results/2024-08-01T11:10:56Z"
+2024/08/01 11:10:56 WARN Differences found.
+
+Applying helper resources for fetching results
+volumesnapshot.snapshot.storage.k8s.io/shoot-comparator-pvc created
+persistentvolumeclaim/shoot-comparator-pvc-read-only created
+pod/fetch-test-comparison-results created
+
+Waiting for fetch-test-comparison-results pod to be ready
+pod/fetch-test-comparison-results condition met
+fetch-test-comparison-results pod is ready
+Copying comparison results to /tmp/shoot_compare
+```
+
+Please mind that volume operations such as snapshot creation and PVC creation may take some time. The job will be started only after the volumes are ready.
+
+In case timeout occurs when waiting for the job to complete, the script will exit with the following message:
+```
+Waiting for the job to complete. It may take couple of minutes. Please, be patient!
+error: timed out waiting for the condition on jobs/compare-shoots
+Job is still not completed. Please check it manually. Exiting...
+```
+> Note: mind the script creates additional resources in the `kcp-system` namespace. Once you are done with the comparison, you can clean up the resources by executing the following command:
+> ```bash
+> ./cleanup-comparison-resources.sh
+> ```
+
+In such case you can check the job status manually by executing the following command:
+```bash
+kubectl get job compare-shoots -n kcp-system
+```
+
+In case timeout occurs when waiting for the pod to be ready, the script will exit with the following message:
+```
+Waiting for fetch-test-comparison-results pod to be ready
+error: timed out waiting for the condition on pods/fetch-test-comparison-results
+fetch-test-comparison-results pod is not ready. Please check it manually. Exiting...`
+```
+
+In such case you can check the pod status manually by executing the following commands:
+```bash
+kubectl get po fetch-test-comparison-results -n kcp-system
+kubectl describe po fetch-test-comparison-results -n kcp-system
+```
+
+## Comparing files starting from a specific date
+
+If you want to compare files older than a specific date, you can specify the date in the `./manifests/job.yaml` script. 
+
+## Analyzing comparison results
+
+If any differences were detected you can analyze the results by examining the content of the `result.txt` file stored in the output directory. The file will contain the details of the comparison, such as the names of the files that differ.
+
+The following is an example of the `result.txt` file content:
+```
+Comparing files older than:0001-01-01 00:00:00 +0000 UTC
+
+Number of files in /Users/i326211/dev/temp/kim-test/shoot-comparator/test2/kim directory = 2
+Number of files in /Users/i326211/dev/temp/kim-test/shoot-comparator/test2/provisioner directory = 2
+
+Differences found.
+
+------------------------------------------------------------------------------------------
+Files that differ: 
+- shoot1.yaml
+------------------------------------------------------------------------------------------
+```
+
+In order to fetch the compared files you must copy the contents of the files. You can do it by executing the following command:
+```bash
+./fetch_shoots_for_provisioner_and_kim.sh
+```

hack/shoot-comparator/scripts/cleanup-comparison-resources.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+echo "Cleaning up the resources created for shoot comparison"
+printf "\n"
+
+echo "Removing resources needed for fetching results"
+kubectl delete -n kcp-system po fetch-test-comparison-results --ignore-not-found
+kubectl delete -n kcp-system pvc shoot-comparator-pvc-read-only --ignore-not-found
+kubectl delete -n kcp-system volumesnapshot shoot-comparator-pvc --ignore-not-found
+
+printf "\n"
+
+echo "Removing resources needed for performing comparison"
+kubectl delete -n kcp-system job/compare-shoots --ignore-not-found
+kubectl delete -n kcp-system pvc/shoot-comparator-pvc --ignore-not-found
+
+kubectl delete -n kcp-system pvc test-prov-shoot-read-only test-kim-shoot-read-only --ignore-not-found
+kubectl delete -n kcp-system volumesnapshot test-kim-shoot-spec-storage test-prov-shoot-spec-storage --ignore-not-found
+
+printf "\n"

hack/shoot-comparator/scripts/fetch_shoots_for_provisioner_and_kim.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+echo "Starting pod that mounts KIM, and Provisioner volumes"
+kubectl apply -f ./manifests/fetch-kim-and-provisioner-files.yaml
+printf "\n"
+
+echo "Waiting for fetch-test-shoot-specs pod to be ready"
+kubectl wait --for=condition=ready pod/fetch-test-shoot-specs -n kcp-system --timeout=5m
+
+result=$?
+if (( $result == 0 ))
+then
+  echo "fetch-test-shoot-specs pod is ready"
+else
+  echo "fetch-test-shoot-specs pod is not ready. Please check it manually. Exiting..."
+  exit 1
+fi
+
+printf "\n"
+
+echo "Copying KIM specs to /tmp/shoot_specs/kim"
+kubectl cp kcp-system/fetch-test-shoot-specs:testdata/kim/ /tmp/shoot_specs/kim
+
+echo "Copying Provisioner specs to /tmp/shoot_specs/provisioner"
+kubectl cp kcp-system/fetch-test-shoot-specs:testdata/provisioner/ /tmp/shoot_specs/provisioner
+
+printf "\n"
+
+echo "Cleaning up the pod created for fetching shoots for provisioner and kim"
+kubectl delete po fetch-test-shoot-specs -n kcp-system --ignore-not-found

hack/shoot-comparator/scripts/manifests/fetch-kim-and-provisioner-files.yaml

@@ -0,0 +1,34 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  labels:
+    run: fetch-test-shoot-specs
+  name: fetch-test-shoot-specs
+  namespace: kcp-system
+spec:
+  volumes:
+  - name: provisioner-ps
+    persistentVolumeClaim:
+      claimName: test-prov-shoot-read-only
+      readOnly: true
+  - name: provisioner-pk
+    persistentVolumeClaim:
+      claimName: test-kim-shoot-read-only
+      readOnly: true
+  containers:
+  - args:
+    - /bin/sh
+    - -c
+    - while true; do date; sleep 3; done
+    image: europe-docker.pkg.dev/kyma-project/prod/external/busybox:1.34.1-v1
+    name: get-pvc
+    volumeMounts:
+    - name: provisioner-ps
+      mountPath: /testdata/provisioner
+      readOnly: true
+    - name: provisioner-pk
+      mountPath: /testdata/kim
+      readOnly: true
+    resources: {}
+  dnsPolicy: ClusterFirst
+  restartPolicy: Always

hack/shoot-comparator/scripts/manifests/fetch-results-pod.yaml

@@ -0,0 +1,55 @@
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshot
+metadata:
+  name: shoot-comparator-pvc
+  namespace: kcp-system
+spec:
+  volumeSnapshotClassName: kim-shoot-dump
+  source:
+    persistentVolumeClaimName: shoot-comparator-pvc
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: shoot-comparator-pvc-read-only
+  namespace: kcp-system
+spec:
+  storageClassName: standard-rwo
+  dataSource:
+    apiGroup: snapshot.storage.k8s.io
+    kind: VolumeSnapshot
+    name: shoot-comparator-pvc
+  accessModes:
+    - ReadOnlyMany
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  labels:
+    run: fetch-test-comparison-results
+  name: fetch-test-comparison-results
+  namespace: kcp-system
+spec:
+  volumes:
+  - name: shoot-comparison-results
+    persistentVolumeClaim:
+      claimName: shoot-comparator-pvc-read-only
+      readOnly: true
+  containers:
+  - args:
+    - /bin/sh
+    - -c
+    - while true; do date; sleep 3; done
+    image: europe-docker.pkg.dev/kyma-project/prod/external/busybox:1.34.1-v1
+    name: get-pvc
+    volumeMounts:
+    - name: shoot-comparison-results
+      mountPath: /results
+      readOnly: true
+    resources: {}
+  dnsPolicy: ClusterFirst
+  restartPolicy: Always
+status: {}

hack/shoot-comparator/scripts/manifests/job.yaml

@@ -0,0 +1,64 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: shoot-comparator-pvc
+  namespace: kcp-system
+spec:
+  storageClassName: standard-rwo
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: compare-shoots
+  namespace: kcp-system
+spec:
+  template:
+    metadata:
+      annotations:
+        sidecar.istio.io/inject: "false"
+    spec:
+      securityContext:
+        fsGroup: 65532
+      volumes:
+      - name: shoot-comparator-pvc
+        persistentVolumeClaim:
+          claimName: shoot-comparator-pvc
+      - name: provisioner-files
+        persistentVolumeClaim:
+          claimName: test-prov-shoot-read-only
+          readOnly: true
+      - name: kim-files
+        persistentVolumeClaim:
+          claimName: test-kim-shoot-read-only
+          readOnly: true
+      containers:
+      - command:
+        - /shoot-comparator
+        - dirs
+        - /testdata/provisioner
+        - /testdata/kim
+        - --outputDir
+        - /results
+# You can specify a date when the comparison should start from. The date should be in RFC3339 format.
+#        - --fromDate
+#        - 2024-07-31T20:04:29Z
+        image: europe-docker.pkg.dev/kyma-project/dev/shoot-comparator:PR-321
+        name: compare-shoots
+        resources: {}
+        securityContext:
+          runAsUser: 65532
+        volumeMounts:
+        - name: shoot-comparator-pvc
+          mountPath: /results
+        - name: provisioner-files
+          mountPath: /testdata/provisioner
+          readOnly: true
+        - name: kim-files
+          mountPath: /testdata/kim
+          readOnly: true
+      restartPolicy: Never

hack/shoot-comparator/scripts/manifests/snapshot-for-comparison.yaml

@@ -0,0 +1,53 @@
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshot
+metadata:
+  name: test-prov-shoot-spec-storage
+  namespace: kcp-system
+spec:
+  volumeSnapshotClassName: kim-shoot-dump
+  source:
+    persistentVolumeClaimName: test-prov-shoot-spec-storage
+---
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshot
+metadata:
+  name: test-kim-shoot-spec-storage
+  namespace: kcp-system
+spec:
+  volumeSnapshotClassName: kim-shoot-dump
+  source:
+    persistentVolumeClaimName: test-kim-shoot-spec-storage
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: test-prov-shoot-read-only
+  namespace: kcp-system
+spec:
+  storageClassName: standard-rwo
+  dataSource:
+    apiGroup: snapshot.storage.k8s.io
+    kind: VolumeSnapshot
+    name: test-prov-shoot-spec-storage
+  accessModes:
+    - ReadOnlyMany
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: test-kim-shoot-read-only
+  namespace: kcp-system
+spec:
+  storageClassName: standard-rwo
+  dataSource:
+    apiGroup: snapshot.storage.k8s.io
+    kind: VolumeSnapshot
+    name: test-kim-shoot-spec-storage
+  accessModes:
+    - ReadOnlyMany
+  resources:
+    requests:
+      storage: 1Gi