diff --git a/hack/shoot-comparator/scripts/README.md b/hack/shoot-comparator/scripts/README.md new file mode 100644 index 00000000..1232b561 --- /dev/null +++ b/hack/shoot-comparator/scripts/README.md @@ -0,0 +1,126 @@ +# Shoot comparator scripts + +## Overview +The scripts are designed to make shoot comparison easier. + +## Running comparison scripts + +In order to run the comparison scripts, execute the following command: +```bash +./run-comparison.sh +``` + +The script will perform the following steps: +- Invoke ./cleanup-comparison-resources.sh to clean up the resources created during the previous comparison +- Prepare volume snapshots and PVCs containing files stored by Provisioner and KIM +- Run the comparison job that mounts the PVCs and compares the files. Results will be stored in the separate volume. +- Prepare volume snapshot and PVC containing the comparison results +- Run pod that mounts the PVC +- Copy the comparison results to the local directory + +The following is an example of the script output: +``` +Cleaning up the resources created for shoot comparison + +Removing resources needed for fetching results +pod "fetch-test-comparison-results" deleted +persistentvolumeclaim "shoot-comparator-pvc-read-only" deleted +volumesnapshot.snapshot.storage.k8s.io "shoot-comparator-pvc" deleted + +Removing resources needed for performing comparison +job.batch "compare-shoots" deleted +persistentvolumeclaim "shoot-comparator-pvc" deleted +persistentvolumeclaim "test-prov-shoot-read-only" deleted +persistentvolumeclaim "test-kim-shoot-read-only" deleted +volumesnapshot.snapshot.storage.k8s.io "test-kim-shoot-spec-storage" deleted +volumesnapshot.snapshot.storage.k8s.io "test-prov-shoot-spec-storage" deleted + +Preparing data for comparison +volumesnapshot.snapshot.storage.k8s.io/test-prov-shoot-spec-storage created +volumesnapshot.snapshot.storage.k8s.io/test-kim-shoot-spec-storage created +persistentvolumeclaim/test-prov-shoot-read-only created +persistentvolumeclaim/test-kim-shoot-read-only created + +Running comparison job +persistentvolumeclaim/shoot-comparator-pvc created +job.batch/compare-shoots created + +Waiting for the job to complete. It may take couple of minutes. Please, be patient! +job.batch/compare-shoots condition met +Job completed + +Fetching logs for the job +2024/08/01 11:10:56 INFO Comparing directories: /testdata/provisioner and /testdata/kim +2024/08/01 11:10:56 INFO Saving comparison details +2024/08/01 11:10:56 INFO Results stored in "/results/2024-08-01T11:10:56Z" +2024/08/01 11:10:56 WARN Differences found. + +Applying helper resources for fetching results +volumesnapshot.snapshot.storage.k8s.io/shoot-comparator-pvc created +persistentvolumeclaim/shoot-comparator-pvc-read-only created +pod/fetch-test-comparison-results created + +Waiting for fetch-test-comparison-results pod to be ready +pod/fetch-test-comparison-results condition met +fetch-test-comparison-results pod is ready +Copying comparison results to /tmp/shoot_compare +``` + +Please mind that volume operations such as snapshot creation and PVC creation may take some time. The job will be started only after the volumes are ready. + +In case timeout occurs when waiting for the job to complete, the script will exit with the following message: +``` +Waiting for the job to complete. It may take couple of minutes. Please, be patient! +error: timed out waiting for the condition on jobs/compare-shoots +Job is still not completed. Please check it manually. Exiting... +``` +> Note: mind the script creates additional resources in the `kcp-system` namespace. Once you are done with the comparison, you can clean up the resources by executing the following command: +> ```bash +> ./cleanup-comparison-resources.sh +> ``` + +In such case you can check the job status manually by executing the following command: +```bash +kubectl get job compare-shoots -n kcp-system +``` + +In case timeout occurs when waiting for the pod to be ready, the script will exit with the following message: +``` +Waiting for fetch-test-comparison-results pod to be ready +error: timed out waiting for the condition on pods/fetch-test-comparison-results +fetch-test-comparison-results pod is not ready. Please check it manually. Exiting...` +``` + +In such case you can check the pod status manually by executing the following commands: +```bash +kubectl get po fetch-test-comparison-results -n kcp-system +kubectl describe po fetch-test-comparison-results -n kcp-system +``` + +## Comparing files starting from a specific date + +If you want to compare files older than a specific date, you can specify the date in the `./manifests/job.yaml` script. + +## Analyzing comparison results + +If any differences were detected you can analyze the results by examining the content of the `result.txt` file stored in the output directory. The file will contain the details of the comparison, such as the names of the files that differ. + +The following is an example of the `result.txt` file content: +``` +Comparing files older than:0001-01-01 00:00:00 +0000 UTC + +Number of files in /Users/i326211/dev/temp/kim-test/shoot-comparator/test2/kim directory = 2 +Number of files in /Users/i326211/dev/temp/kim-test/shoot-comparator/test2/provisioner directory = 2 + +Differences found. + +------------------------------------------------------------------------------------------ +Files that differ: +- shoot1.yaml +------------------------------------------------------------------------------------------ +``` + +In order to fetch the compared files you must copy the contents of the files. You can do it by executing the following command: +```bash +./fetch_shoots_for_provisioner_and_kim.sh +``` diff --git a/hack/shoot-comparator/scripts/cleanup-comparison-resources.sh b/hack/shoot-comparator/scripts/cleanup-comparison-resources.sh new file mode 100755 index 00000000..82170a13 --- /dev/null +++ b/hack/shoot-comparator/scripts/cleanup-comparison-resources.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +echo "Cleaning up the resources created for shoot comparison" +printf "\n" + +echo "Removing resources needed for fetching results" +kubectl delete -n kcp-system po fetch-test-comparison-results --ignore-not-found +kubectl delete -n kcp-system pvc shoot-comparator-pvc-read-only --ignore-not-found +kubectl delete -n kcp-system volumesnapshot shoot-comparator-pvc --ignore-not-found + +printf "\n" + +echo "Removing resources needed for performing comparison" +kubectl delete -n kcp-system job/compare-shoots --ignore-not-found +kubectl delete -n kcp-system pvc/shoot-comparator-pvc --ignore-not-found + +kubectl delete -n kcp-system pvc test-prov-shoot-read-only test-kim-shoot-read-only --ignore-not-found +kubectl delete -n kcp-system volumesnapshot test-kim-shoot-spec-storage test-prov-shoot-spec-storage --ignore-not-found + +printf "\n" diff --git a/hack/shoot-comparator/scripts/fetch_shoots_for_provisioner_and_kim.sh b/hack/shoot-comparator/scripts/fetch_shoots_for_provisioner_and_kim.sh new file mode 100755 index 00000000..e4590701 --- /dev/null +++ b/hack/shoot-comparator/scripts/fetch_shoots_for_provisioner_and_kim.sh @@ -0,0 +1,29 @@ +#!/bin/bash +echo "Starting pod that mounts KIM, and Provisioner volumes" +kubectl apply -f ./manifests/fetch-kim-and-provisioner-files.yaml +printf "\n" + +echo "Waiting for fetch-test-shoot-specs pod to be ready" +kubectl wait --for=condition=ready pod/fetch-test-shoot-specs -n kcp-system --timeout=5m + +result=$? +if (( $result == 0 )) +then + echo "fetch-test-shoot-specs pod is ready" +else + echo "fetch-test-shoot-specs pod is not ready. Please check it manually. Exiting..." + exit 1 +fi + +printf "\n" + +echo "Copying KIM specs to /tmp/shoot_specs/kim" +kubectl cp kcp-system/fetch-test-shoot-specs:testdata/kim/ /tmp/shoot_specs/kim + +echo "Copying Provisioner specs to /tmp/shoot_specs/provisioner" +kubectl cp kcp-system/fetch-test-shoot-specs:testdata/provisioner/ /tmp/shoot_specs/provisioner + +printf "\n" + +echo "Cleaning up the pod created for fetching shoots for provisioner and kim" +kubectl delete po fetch-test-shoot-specs -n kcp-system --ignore-not-found \ No newline at end of file diff --git a/hack/shoot-comparator/scripts/manifests/fetch-kim-and-provisioner-files.yaml b/hack/shoot-comparator/scripts/manifests/fetch-kim-and-provisioner-files.yaml new file mode 100644 index 00000000..82266ec6 --- /dev/null +++ b/hack/shoot-comparator/scripts/manifests/fetch-kim-and-provisioner-files.yaml @@ -0,0 +1,34 @@ +apiVersion: v1 +kind: Pod +metadata: + labels: + run: fetch-test-shoot-specs + name: fetch-test-shoot-specs + namespace: kcp-system +spec: + volumes: + - name: provisioner-ps + persistentVolumeClaim: + claimName: test-prov-shoot-read-only + readOnly: true + - name: provisioner-pk + persistentVolumeClaim: + claimName: test-kim-shoot-read-only + readOnly: true + containers: + - args: + - /bin/sh + - -c + - while true; do date; sleep 3; done + image: europe-docker.pkg.dev/kyma-project/prod/external/busybox:1.34.1-v1 + name: get-pvc + volumeMounts: + - name: provisioner-ps + mountPath: /testdata/provisioner + readOnly: true + - name: provisioner-pk + mountPath: /testdata/kim + readOnly: true + resources: {} + dnsPolicy: ClusterFirst + restartPolicy: Always \ No newline at end of file diff --git a/hack/shoot-comparator/scripts/manifests/fetch-results-pod.yaml b/hack/shoot-comparator/scripts/manifests/fetch-results-pod.yaml new file mode 100644 index 00000000..acc74387 --- /dev/null +++ b/hack/shoot-comparator/scripts/manifests/fetch-results-pod.yaml @@ -0,0 +1,55 @@ +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshot +metadata: + name: shoot-comparator-pvc + namespace: kcp-system +spec: + volumeSnapshotClassName: kim-shoot-dump + source: + persistentVolumeClaimName: shoot-comparator-pvc +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: shoot-comparator-pvc-read-only + namespace: kcp-system +spec: + storageClassName: standard-rwo + dataSource: + apiGroup: snapshot.storage.k8s.io + kind: VolumeSnapshot + name: shoot-comparator-pvc + accessModes: + - ReadOnlyMany + resources: + requests: + storage: 1Gi +--- +apiVersion: v1 +kind: Pod +metadata: + labels: + run: fetch-test-comparison-results + name: fetch-test-comparison-results + namespace: kcp-system +spec: + volumes: + - name: shoot-comparison-results + persistentVolumeClaim: + claimName: shoot-comparator-pvc-read-only + readOnly: true + containers: + - args: + - /bin/sh + - -c + - while true; do date; sleep 3; done + image: europe-docker.pkg.dev/kyma-project/prod/external/busybox:1.34.1-v1 + name: get-pvc + volumeMounts: + - name: shoot-comparison-results + mountPath: /results + readOnly: true + resources: {} + dnsPolicy: ClusterFirst + restartPolicy: Always +status: {} diff --git a/hack/shoot-comparator/scripts/manifests/job.yaml b/hack/shoot-comparator/scripts/manifests/job.yaml new file mode 100644 index 00000000..8b14c190 --- /dev/null +++ b/hack/shoot-comparator/scripts/manifests/job.yaml @@ -0,0 +1,64 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: shoot-comparator-pvc + namespace: kcp-system +spec: + storageClassName: standard-rwo + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: compare-shoots + namespace: kcp-system +spec: + template: + metadata: + annotations: + sidecar.istio.io/inject: "false" + spec: + securityContext: + fsGroup: 65532 + volumes: + - name: shoot-comparator-pvc + persistentVolumeClaim: + claimName: shoot-comparator-pvc + - name: provisioner-files + persistentVolumeClaim: + claimName: test-prov-shoot-read-only + readOnly: true + - name: kim-files + persistentVolumeClaim: + claimName: test-kim-shoot-read-only + readOnly: true + containers: + - command: + - /shoot-comparator + - dirs + - /testdata/provisioner + - /testdata/kim + - --outputDir + - /results +# You can specify a date when the comparison should start from. The date should be in RFC3339 format. +# - --fromDate +# - 2024-07-31T20:04:29Z + image: europe-docker.pkg.dev/kyma-project/dev/shoot-comparator:PR-321 + name: compare-shoots + resources: {} + securityContext: + runAsUser: 65532 + volumeMounts: + - name: shoot-comparator-pvc + mountPath: /results + - name: provisioner-files + mountPath: /testdata/provisioner + readOnly: true + - name: kim-files + mountPath: /testdata/kim + readOnly: true + restartPolicy: Never diff --git a/hack/shoot-comparator/scripts/manifests/snapshot-for-comparison.yaml b/hack/shoot-comparator/scripts/manifests/snapshot-for-comparison.yaml new file mode 100644 index 00000000..58c263a9 --- /dev/null +++ b/hack/shoot-comparator/scripts/manifests/snapshot-for-comparison.yaml @@ -0,0 +1,53 @@ +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshot +metadata: + name: test-prov-shoot-spec-storage + namespace: kcp-system +spec: + volumeSnapshotClassName: kim-shoot-dump + source: + persistentVolumeClaimName: test-prov-shoot-spec-storage +--- +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshot +metadata: + name: test-kim-shoot-spec-storage + namespace: kcp-system +spec: + volumeSnapshotClassName: kim-shoot-dump + source: + persistentVolumeClaimName: test-kim-shoot-spec-storage +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: test-prov-shoot-read-only + namespace: kcp-system +spec: + storageClassName: standard-rwo + dataSource: + apiGroup: snapshot.storage.k8s.io + kind: VolumeSnapshot + name: test-prov-shoot-spec-storage + accessModes: + - ReadOnlyMany + resources: + requests: + storage: 1Gi +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: test-kim-shoot-read-only + namespace: kcp-system +spec: + storageClassName: standard-rwo + dataSource: + apiGroup: snapshot.storage.k8s.io + kind: VolumeSnapshot + name: test-kim-shoot-spec-storage + accessModes: + - ReadOnlyMany + resources: + requests: + storage: 1Gi \ No newline at end of file diff --git a/hack/shoot-comparator/scripts/run_comparison.sh b/hack/shoot-comparator/scripts/run_comparison.sh new file mode 100755 index 00000000..0146c351 --- /dev/null +++ b/hack/shoot-comparator/scripts/run_comparison.sh @@ -0,0 +1,58 @@ +#!/bin/bash + +./cleanup-comparison-resources.sh + +echo "Preparing data for comparison" +kubectl apply -f ./manifests/snapshot-for-comparison.yaml + +printf "\n" + +printf "Running comparison job \n" +kubectl apply -f ./manifests/job.yaml + +printf "\n" + +# wait for completion +echo "Waiting for the job to complete. It may take couple of minutes. Please, be patient!" +kubectl wait --for=condition=complete job/compare-shoots -n kcp-system --timeout=10m + +result=$? +if (( $result == 0 )) +then + echo "Job completed" +else + if kubectl wait --for=condition=failed --timeout=0 job/compare-shoots -n kcp-system 2>/dev/null; then + echo "Job failed to complete. Exiting..." + exit 1 + fi + + echo "Job is still not completed. Please check it manually. Exiting..." + exit 2 +fi + +printf "\n" + +echo "Fetching logs for the job" +kubectl logs job/compare-shoots -n kcp-system + +printf "\n" + +echo "Applying helper resources for fetching results" +kubectl apply -f ./manifests/fetch-results-pod.yaml + +printf "\n" + +echo "Waiting for fetch-test-comparison-results pod to be ready" +kubectl wait --for=condition=ready pod/fetch-test-comparison-results -n kcp-system --timeout=5m + +result=$? +if (( $result == 0 )) +then + echo "fetch-test-comparison-results pod is ready" +else + echo "fetch-test-comparison-results pod is not ready. Please check it manually. Exiting..." + exit 3 +fi + +echo "Copying comparison results to /tmp/shoot_compare" +kubectl cp kcp-system/fetch-test-comparison-results:results/ /tmp/shoot_compare \ No newline at end of file