diff --git a/docs/docs/ci-cd-automation/img/testkube01.png b/docs/docs/ci-cd-automation/img/testkube01.png
new file mode 100644
index 0000000000..23956516de
Binary files /dev/null and b/docs/docs/ci-cd-automation/img/testkube01.png differ
diff --git a/docs/docs/ci-cd-automation/img/tracetest01.png b/docs/docs/ci-cd-automation/img/tracetest01.png
new file mode 100644
index 0000000000..79e0dc4b84
Binary files /dev/null and b/docs/docs/ci-cd-automation/img/tracetest01.png differ
diff --git a/docs/docs/ci-cd-automation/img/tracetest02.png b/docs/docs/ci-cd-automation/img/tracetest02.png
new file mode 100644
index 0000000000..7721e65611
Binary files /dev/null and b/docs/docs/ci-cd-automation/img/tracetest02.png differ
diff --git a/docs/docs/ci-cd-automation/img/tracetest03.png b/docs/docs/ci-cd-automation/img/tracetest03.png
new file mode 100644
index 0000000000..40c8a4abe6
Binary files /dev/null and b/docs/docs/ci-cd-automation/img/tracetest03.png differ
diff --git a/docs/docs/ci-cd-automation/img/tracetest04.png b/docs/docs/ci-cd-automation/img/tracetest04.png
new file mode 100644
index 0000000000..42b9a0a16d
Binary files /dev/null and b/docs/docs/ci-cd-automation/img/tracetest04.png differ
diff --git a/docs/docs/ci-cd-automation/testkube-pipeline.mdx b/docs/docs/ci-cd-automation/testkube-pipeline.mdx
index 41bb6e3188..7fd294cfe2 100644
--- a/docs/docs/ci-cd-automation/testkube-pipeline.mdx
+++ b/docs/docs/ci-cd-automation/testkube-pipeline.mdx
@@ -8,6 +8,7 @@ keywords:
- observability
- distributed tracing
- testing
+ - testkube
image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_Thumbnail_14_rsvkmo.jpg
---
@@ -17,22 +18,30 @@ image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_T
## Running Scheduled Trace-based Tests
-[Testkube Tracetest Executor](https://github.com/kubeshop/testkube-executor-tracetest) is a test executor to run Tracetest tests with Testkube.
+[Tracetest](https://app.tracetest.io/) is a testing tool based on [OpenTelemetry](https://opentelemetry.io/) that allows you to test your distributed application. It uses data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior described by your test definitions.
-:::info
-If you are using the latest version of Testkube, the Tracetest Executor will be bundled in Testkube by default. Read more in the Testkube docs, [here](https://docs.testkube.io/test-types/executor-tracetest/).
+[Testkube](https://testkube.io/) is a Kubernetes-native testing framework for Testers and Developers that allows you to automate the executions of your existing testing tools inside your Kubernetes cluster, removing all the complexity from your CI/CD/GitOps pipelines.
-Or, check out the hands-on workshop on YouTube!
+By using the [Testkube Tracetest Executor](https://docs.testkube.io/test-types/executor-tracetest) you can unlock Testkube's capacity in conjunction with Tracetest, and leverage the work you have already done to instrument your services.
-
-
-:::
+
+
+
## Why do we want to run Tracetest with Testkube?
-Tracetest leverages existing OpenTelemetry instrumentation to run assertions against every part of an HTTP transaction.
+Tracetest leverages existing OpenTelemetry instrumentation to run assertions against every part of a distributed transaction.
-By integrating with Testkube you can now add Tracetest to the native CI/CD pipeline in your Kubernetes cluster. It allows you to run scheduled test runs and synthetic tests. All while following the trace-based testing principle and enabling full in-depth assertions against trace data, not just the response.
+By integrating with Testkube you can now add Tracetest to the native CI/CD pipeline in your Kubernetes cluster. It allows you to run scheduled tests and synthetic tests. All while following the trace-based testing principle and enabling full in-depth assertions against trace data, not just the response.
## Infrastructure Overview
@@ -45,178 +54,70 @@ sequenceDiagram
executor CRDs-->>-testkube: Send details
testkube->>+tracetest executor job: Schedules execution
tracetest executor job->>+tracetest executor job: Configure Tracetest CLI
- tracetest executor job->>+tracetest server: Executes the Tracetest test run
- tracetest server->>+instrumented service: Trigger request
- instrumented service-->>-tracetest server: Get response
+ tracetest executor job->>+tracetest: Executes the Tracetest test run
+ tracetest->>+tracetest agent: Publish test run job
+ tracetest agent->>+instrumented service: Trigger request
+ instrumented service-->>-tracetest agent: Get response
instrumented service->>+data store: Send telemetry data
- tracetest server->>+data store: Fetch trace
- data store-->>-tracetest server: Get trace
- tracetest server->>+tracetest server: Run assertions
- tracetest server-->>-tracetest executor job: Return test run results
+ tracetest agent->>+data store: Fetch trace
+ data store-->>-tracetest agent: Get trace
+ tracetest agent-->>-tracetest: Sync and store response and trace
+ tracetest->>+tracetest: Run analyzer and assertions
+ tracetest-->>-tracetest executor job: Return test run results
tracetest executor job-->>-testkube: Return test run results
testkube-->>-testkube client: Send details
```
-## Prerequisites
-
-Make sure you have these three things installed before starting.
-
-1. A running Kubernetes cluster (either locally or in the cloud).
-2. [Kubectl](https://kubernetes.io/docs/tasks/tools/)
-3. [Helm](https://helm.sh/docs/intro/install/)
-
## Quickstart
-Follow these steps to get started.
+This guide will show how to use Testkube alongside Tracetest to run scheduled Trace-based tests in a Kubernetes cluster.
-## 1. Install Testkube CLI
+### Prerequisites
-Install Testkube CLI by following [these instructions](https://docs.testkube.io/getting-started/step1-installing-cli) for your OS.
+**Tracetest account**:
-```bash
-# MacOS example
-brew install testkube
-```
+- Sign up to [`app.tracetest.io`](https://app.tracetest.io) or follow the [get started](/getting-started/installation) docs.
+- Create an [organization](/concepts/organizations) and [environment](/concepts/environments).
+- Deploy the [Tracetest Agent](/configuration/agent) in your cluster.
+- Create an [environment token](/concepts/environment-tokens).
-## 2. Install Testkube in Your Kubernetes Cluster
+**Testkube account**:
-```bash
-testkube init
-```
+- Sign up to [`app.testkube.io`](https://app.testkube.io) or follow the [get started](https://docs.testkube.io/articles/getting-started) docs.
+- Create a Testkube environment.
+- Deploy the [Testkube Agent](https://docs.testkube.io/testkube-pro/articles/installing-agent) in your cluster.
-Confirm that Testkube is running:
+Remember, in your Kubernetes cluster you should have:
-```bash
-kubectl get all -n testkube
-```
-
-```text title="Expected output"
-NAME READY STATUS RESTARTS AGE
-pod/testkube-api-server-8f5cf8b8f-vrpk6 1/1 Running 3 (14m ago) 18m
-pod/testkube-dashboard-584846b754-4sxq8 1/1 Running 0 18m
-pod/testkube-minio-testkube-bd549c85d-lpwbm 1/1 Running 0 18m
-pod/testkube-mongodb-d78699775-cjcd7 1/1 Running 0 18m
-pod/testkube-nats-0 3/3 Running 0 18m
-pod/testkube-nats-box-5b555bc9c4-5mpz5 1/1 Running 0 18m
-pod/testkube-operator-controller-manager-5f84bb4fd4-mjr2l 2/2 Running 0 18m
-
-NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
-service/testkube-api-server ClusterIP 10.96.78.171 8088/TCP 18m
-service/testkube-dashboard ClusterIP 10.96.204.214 8080/TCP 18m
-service/testkube-minio-service-testkube ClusterIP 10.96.85.219 9000/TCP,9090/TCP,9443/TCP 18m
-service/testkube-mongodb ClusterIP 10.96.213.36 27017/TCP 18m
-service/testkube-nats ClusterIP None 4222/TCP,6222/TCP,8222/TCP,7777/TCP,7422/TCP,7522/TCP 18m
-service/testkube-operator-controller-manager-metrics-service ClusterIP 10.96.243.219 8443/TCP 18m
-service/testkube-operator-webhook-service ClusterIP 10.96.106.138 443/TCP 18m
-
-NAME READY UP-TO-DATE AVAILABLE AGE
-deployment.apps/testkube-api-server 1/1 1 1 18m
-deployment.apps/testkube-dashboard 1/1 1 1 18m
-deployment.apps/testkube-minio-testkube 1/1 1 1 18m
-deployment.apps/testkube-mongodb 1/1 1 1 18m
-deployment.apps/testkube-nats-box 1/1 1 1 18m
-deployment.apps/testkube-operator-controller-manager 1/1 1 1 18m
-
-NAME DESIRED CURRENT READY AGE
-replicaset.apps/testkube-api-server-8f5cf8b8f 1 1 1 18m
-replicaset.apps/testkube-dashboard-584846b754 1 1 1 18m
-replicaset.apps/testkube-minio-testkube-bd549c85d 1 1 1 18m
-replicaset.apps/testkube-mongodb-d78699775 1 1 1 18m
-replicaset.apps/testkube-nats-box-5b555bc9c4 1 1 1 18m
-replicaset.apps/testkube-operator-controller-manager-5f84bb4fd4 1 1 1 18m
-
-NAME READY AGE
-statefulset.apps/testkube-nats 1/1 18m
-```
-
-By default, Testkube is installed in the `testkube` namespace.
+1. `Tracetest Agent`.
+2. `Testkube Agent`.
+3. `OpenTelemetry Instrumented Service`: In order to generate traces and spans, the service under test must support the basics for [propagation](https://opentelemetry.io/docs/reference/specification/context/api-propagators/) through HTTP requests, and also store traces and spans into a Tracing Backend (Jaeger, Grafana Tempo, OpenSearch, etc) or use the [OpenTelemetry Collector](https://docs.tracetest.io/configuration/overview#using-tracetest-without-a-trace-data-store). If you are using a Tracing Backend, the Tracetest Agent requires network access to it. In case of using the OpenTelemetry Collector, you need to setup your collector to send trace data to the Tracetest Agent.
-To explore the Testkube dashboard, run the command:
+On your machine you should have:
-```bash
-testkube dashboard
-```
+1. `Kubectl` [installed](https://kubernetes.io/docs/tasks/tools/).
+2. `Testkube CLI` [installed](https://docs.testkube.io/articles/install/cli).
+3. `Helm` [installed](https://helm.sh/docs/intro/install/)
-
+With everything set up, we will start configuring Testkube and Tracetest to run scheduled Trace-based tests.
-## 3. Install Tracetest CLI
+### 1. Connect your Testkube CLI to your Testkube Environment
-Install Tracetest CLI by following [these instructions](https://docs.tracetest.io/getting-started/installation) for your OS.
+To use the Testkube CLI with your Testkube account you need to set the CLI Context. For that you need a Testkube Pro token. When the token is created, you are ready to change the Testkube CLI context:
```bash
-# MacOS example
-brew install kubeshop/tracetest/tracetest
+testkube set context -c cloud -e testkube-environment-id -o testkube-organization-id -k testkube-token
```
-## 4. Install Tracetest in Your Kubernetes Cluster
+For more information see the Testkube [Connecting from the CLI](https://docs.testkube.io/testkube-pro/articles/managing-cli-context) docs.
-```bash
-tracetest server install
-```
-
-```text title="Expected output"
-How do you want to run TraceTest? [type to search]:
- Using Docker Compose
-> Using Kubernetes
-```
-
-Select `Using Kubernetes`.
-
-```text title="Expected output"
-Do you have OpenTelemetry based tracing already set up, or would you like us to install a demo tracing environment and app? [type to search]:
- I have a tracing environment already. Just install Tracetest
-> Just learning tracing! Install Tracetest, OpenTelemetry Collector and the sample app.
-```
-
-Select `Just learning tracing! Install Tracetest, OpenTelemetry Collector and the sample app.`.
-
-Confirm that Tracetest is running:
-
-```bash
-kubectl get all -n tracetest
-```
+### 2. Create a Test in Tracetest
-```text title="Expected output"
-NAME READY STATUS RESTARTS AGE
-pod/otel-collector-7f4d87489f-vp6zn 1/1 Running 0 5m41s
-pod/tracetest-78b9c84c57-t4prx 1/1 Running 3 (4m15s ago) 5m29s
-pod/tracetest-postgresql-0 1/1 Running 0 5m42s
-
-NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
-service/otel-collector ClusterIP 10.96.173.226 4317/TCP 5m46s
-service/tracetest ClusterIP 10.96.248.146 11633/TCP,4317/TCP 5m42s
-service/tracetest-postgresql ClusterIP 10.96.155.147 5432/TCP 5m42s
-service/tracetest-postgresql-hl ClusterIP None 5432/TCP 5m42s
-
-NAME READY UP-TO-DATE AVAILABLE AGE
-deployment.apps/otel-collector 1/1 1 1 5m46s
-deployment.apps/tracetest 1/1 1 1 5m42s
-
-NAME DESIRED CURRENT READY AGE
-replicaset.apps/otel-collector-7f4d87489f 1 1 1 5m46s
-replicaset.apps/tracetest-78b9c84c57 1 1 1 5m42s
-
-NAME READY AGE
-statefulset.apps/tracetest-postgresql 1/1 5m42s
-```
+In the environment in your [Tracetest](https://app.tracetest.io) account, start by clicking `Tests` > `Create` > `HTTP`, then `Enter the URL of your OpenTelemetry Instrumented Service` and click `Run`. Remember that you need the Tracetest Agent deployed and configured in your environment so it can trigger the request and fetch the trace data.
-By default, Tracetest is installed in the `tracetest` namespace.
+This will trigger the test and display the distributed trace in the `Test` tab to run assertions against.
-To explore the Tracetest Web UI, run the command:
-
-```bash
-kubectl --kubeconfig /.kube/config --context --namespace tracetest port-forward svc/tracetest 11633
-```
-
-
-
-## 5. Create a Test in Tracetest
-
-Start by clicking `Create` > `Create New Test` > `HTTP Request` > `Next` > `Choose Example` (dropdown) > `Pokeshop - List` (generates a sample test from the Tracetest demo) > `Next` > `URL` is prefilled with `http://demo-pokemon-api.demo/pokemon?take=20&skip=0` > `Create & Run`.
-
-This will trigger the test and display a distributed trace in the `Trace` tab to run assertions against.
-
-
+
Proceed to add a test spec to assert all database queries return within 500 ms. Click the `Test` tab and proceed to click the `Add Test Spec` button.
@@ -229,16 +130,16 @@ span[tracetest.span.type="database"]
In the assertion field add:
```css
-attr:tracetest.span.duration < 500ms
+attr: tracetest.span.duration < 500ms;
```
Save the test spec and publish the test.
-
+
-The database spans that are returning in more than 500ms are labeled in red.
+The database spans that are returning in less than 500ms are labeled in green.
-
+
This is an example of a trace-based test that asserts against every single part of an HTTP transaction, including all interactions with the database.
@@ -246,29 +147,13 @@ However, Tracetest cannot run this test as part of your CI/CD without integratin
Let's introduce how Testkube makes it possible.
-## 6. Deploy the Tracetest Executor
-
-Testkube works with the concept of Executors. An Executor is a wrapper around a testing framework, Tracetest in this case, in the form of a Docker container and runs as a Kubernetes job. To start you need to register and deploy the Tracetest executor in your cluster using the Testkube CLI.
-
-```bash
-kubectl testkube create executor --image kubeshop/testkube-executor-tracetest:latest --types "tracetest/test" --name tracetest-executor --icon-uri icon --content-type string --content-type file-uri
-```
-
-```text title="Expected output"
-Executor created tracetest-executor π₯
-```
-
-## 7. Create a Trace-based Test with Tracetest in Testkube
+### 3. Create a Trace-based Test with Tracetest in Testkube
-Click the βοΈ button in the top right. Then click `Test Definition`.
+Click the `Automate` tab in the top of your screen. This will show the YAML definition for the test run.
-
+
-This will open a YAML definition for the test run.
-
-
-
-Save this into a file called `test.yaml`:
+Click the `Download File` button and this will save your definition in a yaml format.
```yaml
type: Test
@@ -282,21 +167,23 @@ spec:
url: http://demo-pokemon-api.demo/pokemon?take=20&skip=0
method: GET
headers:
- - key: Content-Type
- value: application/json
+ - key: Content-Type
+ value: application/json
specs:
- - name: Database queries less than 500 ms
- selector: span[tracetest.span.type="database"]
- assertions:
- - attr:tracetest.span.duration < 500ms
+ - name: Database queries less than 500 ms
+ selector: span[tracetest.span.type="database"]
+ assertions:
+ - attr:tracetest.span.duration < 500ms
```
-Execute the following command to create the test executor object in Testkube. Do not forget to provide the path to your Tracetest definition file using the `--file` argument and also the Tracetest Server endpoint using the `TRACETEST_ENDPOINT` `--variable`.
+Execute the following command to create the test executor object in Testkube. Do not forget to provide the path to your Tracetest definition file using the `--file` argument, and also the following variables:
-Remember that your `TRACETEST_ENDPOINT` should be reachable from Testkube in your cluster. Use your Tracetest service's `CLUSTER-IP:PORT`. E.g: `10.96.93.106:11633`.
+- `TRACETEST_TOKEN`: your [environment token](../concepts/environment-token.mdx).
+- `TRACETEST_ENVIRONMENT`: your [environment id](../concepts/environments.mdx).
+- `TRACETEST_ORGANIZATION`: your [organization id](../concepts/organizations.mdx).
```bash
-kubectl testkube create test --file ./test.yaml --type "tracetest/test" --name pokeshop-tracetest-test --variable TRACETEST_ENDPOINT=http://CLUSTER-IP:PORT
+kubectl testkube create test --file my/file/location.yaml --type "tracetest/test" --name pokeshop-tracetest-test --variable TRACETEST_TOKEN=tracetest-token --variable TRACETEST_ENVIRONMENT=tracetest-environment-id --variable TRACETEST_ORGANIZATION=tracetest-organization-id
```
```text title="Expected output"
@@ -307,7 +194,7 @@ Opening the Testkube Dashboard will show the test is created successfully.
-## 8. Run the Tracetest Trace-based Test in Testkube
+### 4. Run the Tracetest Trace-based Test in Testkube
Finally, to run the test, execute the following command, or run the test from the Testkube Dashboard.
@@ -315,87 +202,40 @@ Finally, to run the test, execute the following command, or run the test from th
kubectl testkube run test --watch pokeshop-tracetest-test
```
-Here's what the Testkube CLI will look like if the test fails.
+Here's what the Testkube CLI will look like if the test passes.
-```text title="Expected output"
-Type: tracetest/test
-Name: pokeshop-tracetest-test
-Execution ID: 641885f39922b3e1003dd5b6
-Execution name: pokeshop-tracetest-test-3
-Execution number: 3
-Status: running
-Start time: 2023-03-20 16:12:35.268197087 +0000 UTC
-End time: 0001-01-01 00:00:00 +0000 UTC
-Duration:
-
- Variables: 1
- - TRACETEST_ENDPOINT = http://10.96.93.106:11633
-
-Getting logs from test job 641885f39922b3e1003dd5b6
-Execution completed
+```bash
+π [TracetestRunner]: Preparing test run
+π Configuring Tracetest CLI with Token
+π Using arguments to configure CLI: [configure --token tracetest-token --organization tracetest-organization-id --environment tracetest-environment-id]
+π Configure command tracetest configure --token tracetest-token --organization tracetest-organization-id --environment tracetest-environment-id
π¬ Executing in directory :
- $ tracetest run test --server-url http://10.96.93.106:11633 --file /tmp/test-content737616681 --output pretty
-β Pokeshop - List (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test)
- β Database queries less than 500 ms
- β #2b213392d0e3ff21
- β attr:tracetest.span.duration < 500ms (502ms) (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test?selectedAssertion=0&selectedSpan=2b213392d0e3ff21)
- β #7e6657f6a43fceeb
- β attr:tracetest.span.duration < 500ms (72ms)
- β #6ee2fb69690eed47
- β attr:tracetest.span.duration < 500ms (13ms)
- β #a82c304a3558763b
- β attr:tracetest.span.duration < 500ms (679ms) (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test?selectedAssertion=0&selectedSpan=a82c304a3558763b)
- β #6ae21f2251101fd6
- β attr:tracetest.span.duration < 500ms (393ms)
- β #2a9b9422af8ba1a8
- β attr:tracetest.span.duration < 500ms (61ms)
- β #010a8a0d53687276
- β attr:tracetest.span.duration < 500ms (36ms)
- β #895d66286b6325ae
- β attr:tracetest.span.duration < 500ms (686ms) (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test?selectedAssertion=0&selectedSpan=895d66286b6325ae)
-
-```
-
-And, here's the Testkube Dashboard.
-
+ $ tracetest configure --token tracetest-token --organization tracetest-organization-id --environment tracetest-environment-id
+ SUCCESS Successfully configured Tracetest CLI
-If the test passes, it'll look like this.
-
-```text title="Expected output"
-Type: tracetest/test
-Name: pokeshop-tracetest-test
-Execution ID: 6418873d9922b3e1003dd5b8
-Execution name: pokeshop-tracetest-test-4
-Execution number: 4
-Status: running
-Start time: 2023-03-20 16:18:05.60245717 +0000 UTC
-End time: 0001-01-01 00:00:00 +0000 UTC
-Duration:
-
- Variables: 1
- - TRACETEST_ENDPOINT = http://10.96.93.106:11633
-
-
-Getting logs from test job 6418873d9922b3e1003dd5b8
-Execution completed
+β Execution succeeded
+π Using arguments to run test: [run test --file /data/test-content --output pretty]
+π Test run command tracetest run test --file /data/test-content --output pretty
π¬ Executing in directory :
- $ tracetest run test --server-url http://10.96.93.106:11633 --file /tmp/test-content1901459587 --output pretty
-β Pokeshop - List (http://10.96.93.106:11633/test/RUkKQ_aVR/run/3/test)
- β Database queries less than 500 ms
+ $ tracetest run test --file /data/test-content --output pretty
-β Execution succeeded
-Execution completed β Pokeshop - List (http://10.96.93.106:11633/test/RUkKQ_aVR/run/3/test)
- β Database queries less than 500 ms
+β RunGroup: #58WmDuBSR (https://app.tracetest.io/organizations/tracetest-organization-id/environments/tracetest-environment-id/run/58WmDuBSR)
+ Summary: 1 passed, 0 failed, 0 pending
+ β Pokeshop - List (https://app.tracetest.io/organizations/tracetest-organization-id/environments/tracetest-environment-id/test/R5NITR14g/run/1/test) - trace id: 3cdcb56d6c226f7083f45d6b3d278051
+ β span[tracetest.span.type="http"]
+ β span[tracetest.span.type="database"]
```
-
+And, here's the Testkube Dashboard.
+
+
-## 9. Create a Trace-based Test That Runs Every Minute
+### 5. Create a Trace-based Test That Runs Every Minute
By using Testkube's [scheduling](https://docs.testkube.io/concepts/scheduling), you can trigger this test every minute.
```bash
-kubectl testkube create test --file ./test.yaml --type "tracetest/test" --name pokeshop-tracetest-scheduled-test --schedule="*/1 * * * *" --variable TRACETEST_ENDPOINT=http://CLUSTER-IP:PORT
+kubectl testkube create test --file my/file/location.yaml --type "tracetest/test" --name pokeshop-tracetest-scheduled-test --schedule="*/1 * * * *" --variable TRACETEST_TOKEN=tracetest-token --variable TRACETEST_ENVIRONMENT=tracetest-environment-id --variable TRACETEST_ORGANIZATION=tracetest-organization-id
```
```text title="Expected output"
@@ -406,6 +246,10 @@ In your Testkube Dashboard you'll see this test run continuously and get trigger
-## Next Steps
+## What's Next?
To explore more options that Testkube gives you, check out [test triggers](https://docs.testkube.io/concepts/triggers). They enable you to trigger tests based on Kubernetes events.
+
+## Learn More
+
+Please visit our [documentation](https://docs.tracetest.io/) and join our [Slack Community](https://dub.sh/tracetest-community) for more info!
diff --git a/docs/docs/concepts/run-groups.md b/docs/docs/concepts/run-groups.md
new file mode 100644
index 0000000000..39024af6ad
--- /dev/null
+++ b/docs/docs/concepts/run-groups.md
@@ -0,0 +1,158 @@
+# Run Groups
+
+Run Groups are containers that encapsulate one or more runs. Each time a test run is executed in the Tracetest app, a run group is generated that can be found in the UI.
+
+
+
+Clicking on a run group will show the test runs the run group contains:
+
+
+
+Run Groups can be used to logically group runs that have context together. This allows specifying a series of tests or test suites to run concurrently. You then wait for the entire collection to pass or fail before making a decision if the entire group succeeded.
+
+:::note
+You can asynchronously run multiple tests from a script in one run group and wait for all of them to finish before deciding whether to deploy a new release.
+:::
+
+Run Groups track the status of all test suites and test runs to yield a single result.
+
+Users are allowed to attach more runs to a run group by adding them in the run information thatβs available in the CLI.
+
+[Integrations](https://docs.tracetest.io/tools-and-integrations/overview) support this feature by creating a single run group for every execution.
+- [k6](https://docs.tracetest.io/tools-and-integrations/k6)
+- [Playwright](https://docs.tracetest.io/tools-and-integrations/playwright)
+- [Cypress](https://docs.tracetest.io/tools-and-integrations/cypress)
+- [Artillery](https://docs.tracetest.io/tools-and-integrations/artillery-plugin)
+
+Run groups are essential for integrations! Without it, a k6 test could potentially generate thousands of test runs that would pollute the environment and make it almost impossible to search for test runs not associated with that k6 test run.
+
+Another good use case for run groups is for CI runs: usually we donβt run a single test or a single test suite to test our systems. You can expect some users to have dozens of suites being run during a CI pipeline. With run groups, all the suite runs from the same pipeline will be shown on the same screen, making it easier for users to understand what failed.
+
+Last, but not least important, if multiple tests are run inside a run group, they will run in parallel and finish faster than if you use a test suite to organize your tests. (Test suites run sequentially, so for the next test to be run, the current test must finish executing, which can be pretty time consuming.)
+
+## Using Run Groups from the CLI
+
+The way to use run groups from the CLI is by running multiple folders at once:
+
+
+
+
+
+Users can also manually wait for a run group to be done from the CLI.
+
+The above screenshots show that now you can run multiple tests at once by providing multiple `-f 0
+```
+
+```
+# test2.yaml
+
+type: Test
+spec:
+ name: Check if there are suites ino the database
+ type: http
+ httpRequest:
+ url: ${env:ENDPOINT}/api/testsuites
+ method: GET
+ specs:
+ - selector: span[name = "Tracetest Trigger"]
+ assertions:
+ - attr:tracetest.response.body | json_path '$.count' > 0
+```
+
+### Running multiple tests inside the same group
+
+```
+tracetest run test -f test1.yaml -f test2.yaml
+```
+
+Both test runs will be shown on the same run group (with a random id).
+
+### Running multiple tests inside the same group and choosing the group id
+
+```
+tracetest run test -f test1.yaml -f test2.yaml --group my-group-id
+```
+
+Both tests will be shown on the same run group (id = my-group-id). If this run group already exists, the two new runs will be placed there with any other test run from that group.
+
+### Adding a new run to an existing group
+
+```
+tracetest run test -f test3.yaml --group my-group-id
+```
+This new run will be part of the `my-group-id` group along with the `test1` and `test2` runs.
+
+### Running a suite and some tests inside the same group
+
+```
+tracetest run test -f suite.yaml -f test1.yaml -f test2.yaml
+```
+
+Test suites can also be part of a run group along with other suites or tests.
+
+### Difference between a test suite and a run group
+
+
+
+Mermaid code (works on GitHub and our doc page) to generate the Gantt graph.
+
+```
+gantt
+ title Test suite vs run group execution
+ dateFormat HH:mm
+ axisFormat %H:%M
+ section Test suite
+ Test 1 :t1, 15:00, 1m
+ Test 2 :t2, after t1 , 2m
+ Test 3 :t3, after t2, 3m
+ Test 4 :t4, after t3, 1m
+
+ section Run Group
+ Test 1 :r1, 15:00, 1m
+ Test 2 :r2, 15:00, 2m
+ Test 3 :r3, 15:00, 3m
+ Test 4 :r4, 15:00, 1m
+```
+
+**Execution time:** A test suite only executes the next test when the current test has finished running. So if it contains 4 tests that take 1 minute each, the whole suite will take 4 minutes to complete. Run group tests are run in parallel, so if the same 4 tests are run inside a group, it will complete in 1 minute.
+
+**Using data from a previous test:** In suites, you can chain your tests and share data from one test to another. For example, getting the response from the first test and use it as part of the second test definition. This is not possible with run groups.
+
+**Composition:** Suites can only contain tests inside of it. Groups can contain both tests and suites. This means that if you have 5 tests where 3 are totally independent but the other 2 need each other to work, you can use groups to run those tests in an efficient way: running the suite (with 2 tests) and the 3 independent tests in parallel. If all tests take 1 minute to run, the run group would take 2 minutes to run.
+
+
diff --git a/docs/docs/concepts/runs.md b/docs/docs/concepts/runs.md
new file mode 100644
index 0000000000..57e2a9f2e9
--- /dev/null
+++ b/docs/docs/concepts/runs.md
@@ -0,0 +1,7 @@
+# Runs
+
+Tracetest uses the concept of Runs to define every time a Test was triggered. Runs from different Tests can be grouped into Run Groups to be triggered in parallel.
+- [Tests](../concepts/tests.md)
+- [Run Groups](../concepts/run-groups.md)
+
+
diff --git a/docs/docs/concepts/tests.md b/docs/docs/concepts/tests.md
new file mode 100644
index 0000000000..d0e6658d2d
--- /dev/null
+++ b/docs/docs/concepts/tests.md
@@ -0,0 +1,18 @@
+# Tests
+
+Tracetest uses the concept of Tests to define how to trigger a test against your application, define assertions against its trace data, and automate its execution. Every time a Test is triggered it will create a Run.
+- [Runs](../concepts/runs.md) (link)
+
+
+
+A test allows you to:
+
+- Execute a trigger, such as an HTTP request, a gRPC call, a TraceID, a Kafka queue, etc. to generate a trace.
+- Fetch the resulting trace and analyze it.
+- Add assertions against the trace data to verify the behavior of the system at every step of the request transaction.
+- The assertions can check things like HTTP status codes, database call durations, gRPC return codes, and other aspects of the distributed system's behavior.
+- Tests can be saved and run manually or as part of a CI/CD pipeline to ensure the quality and reliability of the distributed application.
+
+A test in Tracetest is a way to define trace-based assertions that validate the end-to-end behavior of a distributed system, going beyond just UI or API-level testing.
+
+
diff --git a/docs/docs/img/516718595/runs.png b/docs/docs/img/516718595/runs.png
new file mode 100644
index 0000000000..36579beebe
Binary files /dev/null and b/docs/docs/img/516718595/runs.png differ
diff --git a/docs/docs/img/choose-variable-set-0.16.jpg b/docs/docs/img/choose-variable-set-0.16.jpg
new file mode 100644
index 0000000000..215d05c36c
Binary files /dev/null and b/docs/docs/img/choose-variable-set-0.16.jpg differ
diff --git a/docs/docs/img/run-groups-chart.png b/docs/docs/img/run-groups-chart.png
new file mode 100644
index 0000000000..d359e8b5a9
Binary files /dev/null and b/docs/docs/img/run-groups-chart.png differ
diff --git a/docs/docs/img/run-groups-cli-2.png b/docs/docs/img/run-groups-cli-2.png
new file mode 100644
index 0000000000..61301d2a56
Binary files /dev/null and b/docs/docs/img/run-groups-cli-2.png differ
diff --git a/docs/docs/img/run-groups-cli.png b/docs/docs/img/run-groups-cli.png
new file mode 100644
index 0000000000..96ec81685c
Binary files /dev/null and b/docs/docs/img/run-groups-cli.png differ
diff --git a/docs/docs/img/run-groups-detail.png b/docs/docs/img/run-groups-detail.png
new file mode 100644
index 0000000000..d6d5874808
Binary files /dev/null and b/docs/docs/img/run-groups-detail.png differ
diff --git a/docs/docs/img/run-groups-ui.png b/docs/docs/img/run-groups-ui.png
new file mode 100644
index 0000000000..d50919cbaf
Binary files /dev/null and b/docs/docs/img/run-groups-ui.png differ
diff --git a/docs/docs/img/runs.png b/docs/docs/img/runs.png
new file mode 100644
index 0000000000..36579beebe
Binary files /dev/null and b/docs/docs/img/runs.png differ
diff --git a/docs/docs/img/tests.png b/docs/docs/img/tests.png
new file mode 100644
index 0000000000..6bae43ab1e
Binary files /dev/null and b/docs/docs/img/tests.png differ
diff --git a/docs/docs/tools-and-integrations/img/testkube.gif b/docs/docs/tools-and-integrations/img/testkube.gif
new file mode 100644
index 0000000000..f6c98b8382
Binary files /dev/null and b/docs/docs/tools-and-integrations/img/testkube.gif differ
diff --git a/docs/docs/tools-and-integrations/img/tracetest.gif b/docs/docs/tools-and-integrations/img/tracetest.gif
new file mode 100644
index 0000000000..8a2adbc4d1
Binary files /dev/null and b/docs/docs/tools-and-integrations/img/tracetest.gif differ
diff --git a/docs/docs/tools-and-integrations/testkube.mdx b/docs/docs/tools-and-integrations/testkube.mdx
index 757336b0e6..64b184462e 100644
--- a/docs/docs/tools-and-integrations/testkube.mdx
+++ b/docs/docs/tools-and-integrations/testkube.mdx
@@ -8,29 +8,40 @@ keywords:
- observability
- distributed tracing
- testing
+ - testkube
image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_Thumbnail_14_rsvkmo.jpg
---
-[Tracetest](https://tracetest.io/) is a testing tool based on [OpenTelemetry](https://opentelemetry.io/) that permits you to test your distributed application. It allows you to use the trace data generated by your OpenTelemetry tools to check and assert if your application has the desired behavior defined by your test definitions.
+[Tracetest](https://app.tracetest.io/) is a testing tool based on [OpenTelemetry](https://opentelemetry.io/) that allows you to test your distributed application. It uses data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior described by your test definitions.
[Testkube](https://testkube.io/) is a Kubernetes-native testing framework for Testers and Developers that allows you to automate the executions of your existing testing tools inside your Kubernetes cluster, removing all the complexity from your CI/CD/GitOps pipelines.
-For Testkube, tests are meant to be part of a cluster's state and can be executed as needed:
+## Why is this important?
-- Manually via kubectl CLI.
-- Externally triggered via API (CI, external tooling, etc).
-- Automatically on deployment of annotated/labeled services/pods/etc.
+Integrating Tracetest with Testkube enhances testing processes by leveraging Kubernetes-native capabilities, improving scalability, and reliability of testing workflows for distributed applications.
-By using the [Testkube Tracetest Executor](https://github.com/kubeshop/testkube-executor-tracetest) you can unlock Testkube's capacity in conjunction with Tracetest, and leverage the work you have already done to instrument your services.
+
+
+
-:::info
-If you are using the latest version of Testkube, the Tracetest Executor will be bundled in Testkube by default. Read more in the Testkube docs, [here](https://docs.testkube.io/test-types/executor-tracetest/).
+## The Testkube Tracetest Executor
-Or, check out the hands-on workshop on YouTube!
+For Testkube, tests are meant to be part of a cluster's state and can be executed as needed:
-
+- Manually via kubectl CLI.
+- Externally triggered via API (CI, external tooling, etc).
+- Automatically on deployment of annotated/labeled services/pods/etc.
-:::
+By using the [Testkube Tracetest Executor](https://docs.testkube.io/test-types/executor-tracetest) you can unlock Testkube's capacity in conjunction with Tracetest, and leverage the work you have already done to instrument your services.
## How does it work?
@@ -43,14 +54,16 @@ sequenceDiagram
executor CRDs-->>-testkube: Send details
testkube->>+tracetest executor job: Schedules execution
tracetest executor job->>+tracetest executor job: Configure Tracetest CLI
- tracetest executor job->>+tracetest server: Executes the Tracetest test run
- tracetest server->>+instrumented service: Trigger request
- instrumented service-->>-tracetest server: Get response
+ tracetest executor job->>+tracetest: Executes the Tracetest test run
+ tracetest->>+tracetest agent: Publish test run job
+ tracetest agent->>+instrumented service: Trigger request
+ instrumented service-->>-tracetest agent: Get response
instrumented service->>+data store: Send telemetry data
- tracetest server->>+data store: Fetch trace
- data store-->>-tracetest server: Get trace
- tracetest server->>+tracetest server: Run assertions
- tracetest server-->>-tracetest executor job: Return test run results
+ tracetest agent->>+data store: Fetch trace
+ data store-->>-tracetest agent: Get trace
+ tracetest agent-->>-tracetest: Sync and store response and trace
+ tracetest->>+tracetest: Run analyzer and assertions
+ tracetest-->>-tracetest executor job: Return test run results
tracetest executor job-->>-testkube: Return test run results
testkube-->>-testkube client: Send details
```
@@ -61,34 +74,45 @@ This guide will show how to use Testkube alongside Tracetest to run your tests i
### Prerequisites
-In your Kubernetes cluster you should have:
+**Tracetest account**:
+
+- Sign up to [`app.tracetest.io`](https://app.tracetest.io) or follow the [get started](/getting-started/installation) docs.
+- Create an [organization](/concepts/organizations) and [environment](/concepts/environments).
+- Deploy the [Tracetest Agent](/configuration/agent) in your cluster.
+- Create an [environment token](/concepts/environment-tokens).
+
+**Testkube account**:
-1. `Testkube`: Use HELM or the Testkube CLI to [install](https://kubeshop.github.io/testkube/installing) Testkube Server components in your cluster.
-2. `Tracetest`: You need a running instance of [Tracetest](https://docs.tracetest.io/getting-started/installation/) or [Tracetest Core](https://docs.tracetest.io/core/getting-started/installation/) which is going to be executing your assertions. To do so, you can follow the instructions defined in the Tracetest [documentation](https://docs.tracetest.io/getting-started/overview).
-3. `OpenTelemetry Instrumented Service`: In order to generate traces and spans, the service under test must support the basics for [propagation](https://opentelemetry.io/docs/reference/specification/context/api-propagators/) through HTTP requests, and also store traces and spans into a Data Store Backend (Jaeger, Grafana Tempo, OpenSearch, etc) or use the [OpenTelemetry Collector](https://docs.tracetest.io/configuration/overview#using-tracetest-without-a-trace-data-store).
+- Sign up to [`app.testkube.io`](https://app.testkube.io) or follow the [get started](https://docs.testkube.io/articles/getting-started) docs.
+- Create a Testkube environment.
+- Deploy the [Testkube Agent](https://docs.testkube.io/testkube-pro/articles/installing-agent) in your cluster.
+
+Remember, in your Kubernetes cluster you should have:
+
+1. `Tracetest Agent`.
+2. `Testkube Agent`.
+3. `OpenTelemetry Instrumented Service`: In order to generate traces and spans, the service under test must support the basics for [propagation](https://opentelemetry.io/docs/reference/specification/context/api-propagators/) through HTTP requests, and also store traces and spans into a Tracing Backend (Jaeger, Grafana Tempo, OpenSearch, etc) or use the [OpenTelemetry Collector](https://docs.tracetest.io/configuration/overview#using-tracetest-without-a-trace-data-store). If you are using a Tracing Backend, the Tracetest Agent requires network access to it. In case of using the OpenTelemetry Collector, you need to setup your collector to send trace data to the Tracetest Agent.
On your machine you should have:
1. `Kubectl` [installed](https://kubernetes.io/docs/tasks/tools/).
-2. `Testkube CLI` [installed](https://kubeshop.github.io/testkube/installing#1-installing-the-testkube-cli).
+2. `Testkube CLI` [installed](https://docs.testkube.io/articles/install/cli).
With everything set up, we will start configuring Testkube and Tracetest.
-### 1. Deploy the Tracetest Executor
+### 1. Connect your Testkube CLI to your Testkube Environment
-Testkube works with the concept of [Executors](https://kubeshop.github.io/testkube/test-types/executor-custom). An Executor is a wrapper around a testing framework (Tracetest in this case) in the form of a Docker container and runs as a Kubernetes job. To start, you need to register and deploy the Tracetest executor in your cluster using the Testkube CLI:
+To use the Testkube CLI with your Testkube account you need to set the CLI Context. For that you need a Testkube Pro token. When the token is created, you are ready to change the Testkube CLI context:
```bash
-kubectl testkube create executor --image kubeshop/testkube-executor-tracetest:latest --types "tracetest/test" --name tracetest-executor --icon-uri icon --content-type string --content-type file-uri
+testkube set context -c cloud -e testkube-environment-id -o testkube-organization-id -k testkube-token
```
-:::info
-If you are using the latest version of Testkube, the Tracetest Executor will be bundled in Testkube by default. Read more in the Testkube docs, [here](https://docs.testkube.io/test-types/executor-tracetest/).
-:::
+For more information see the Testkube [Connecting from the CLI](https://docs.testkube.io/testkube-pro/articles/managing-cli-context) docs.
### 2. Create a Test
-Now you need a Tracetest test. Have a look at the [Tracetest documentation](https://docs.tracetest.io/cli/creating-tests) for details on writing tests. Here is a simple test definition example:
+For this step you need a Tracetest test. Have a look at the [Tracetest documentation](/cli/creating-tests) for details on writing tests. Here is a simple test definition example:
```yaml
type: Test
@@ -113,25 +137,62 @@ spec:
- attr:db.name = "pokeshop"
```
-Execute the following command to create the test executor object in Testkube. Do not forget to provide the path to your Tracetest definition file using the `--file` argument, and also the follow variables:
-- **If you are using [Tracetest Core](https://docs.tracetest.io/core/getting-started/installation/)**: Define the Tracetest Server endpoint using the `TRACETEST_ENDPOINT` `--variable` (remember that the TRACETEST_ENDPOINT should be reachable from the Testkube instance).
+Execute the following command to create the test executor object in Testkube. Do not forget to provide the path to your Tracetest definition file using the `--file` argument, and also the following variables:
+
+- `TRACETEST_TOKEN`: your [environment token](../concepts/environment-token.mdx).
+- `TRACETEST_ENVIRONMENT`: your [environment id](../concepts/environments.mdx).
+- `TRACETEST_ORGANIZATION`: your [organization id](../concepts/organizations.mdx).
```bash
-kubectl testkube create test --file my/file/location.yaml --type "tracetest/test" --name pokeshop-tracetest-test --variable TRACETEST_ENDPOINT=http://tracetest
+kubectl testkube create test --file my/file/location.yaml --type "tracetest/test" --name pokeshop-tracetest-test --variable TRACETEST_TOKEN=tracetest-token --variable TRACETEST_ENVIRONMENT=tracetest-environment-id --variable TRACETEST_ORGANIZATION=tracetest-organization-id
```
-Note: In case you are doing a port forward to your Tracetest instance and you want to have the correct Tracetest URL in your results printed by the Testkube output, you can also provide an optional `TRACETEST_OUTPUT_ENDPOINT` variable (e.g. `--variable TRACETEST_OUTPUT_ENDPOINT=http://localhost:11633`).
+### 3. Run the Test
-- **If you are using [Tracetest](https://docs.tracetest.io/getting-started/installation/) on `app.tracetest.io`**: Define your [token](../concepts/environment-token.mdx), [environment](../concepts/environments.mdx) and [organization](../concepts/organizations.mdx) using the `TRACETEST_TOKEN`, `TRACETEST_ENVIRONMENT` and `TRACETEST_ORGANIZATION` variables
+To see the integration working, run the test by executing the following command:
```bash
-kubectl testkube create test --file my/file/location.yaml --type "tracetest/test" --name pokeshop-tracetest-test --variable TRACETEST_TOKEN=your-token --variable TRACETEST_ENVIRONMENT=your-environment-id --variable TRACETEST_ORGANIZATION=your-organization-id
+kubectl testkube run test --watch pokeshop-tracetest-test
```
-### 3. Run the Test
+### 4. Explore the results
-Finally, to see the integration working, run the test by executing the following command:
+Finally, you can see the Tracetest results in your CLI:
```bash
-kubectl testkube run test --watch pokeshop-tracetest-test
+π [TracetestRunner]: Preparing test run
+π Configuring Tracetest CLI with Token
+π Using arguments to configure CLI: [configure --token tracetest-token --organization tracetest-organization-id --environment tracetest-environment-id]
+π Configure command tracetest configure --token tracetest-token --organization tracetest-organization-id --environment tracetest-environment-id
+π¬ Executing in directory :
+ $ tracetest configure --token tracetest-token --organization tracetest-organization-id --environment tracetest-environment-id
+ SUCCESS Successfully configured Tracetest CLI
+
+β Execution succeeded
+π Using arguments to run test: [run test --file /data/test-content --output pretty]
+π Test run command tracetest run test --file /data/test-content --output pretty
+π¬ Executing in directory :
+ $ tracetest run test --file /data/test-content --output pretty
+
+β RunGroup: #58WmDuBSR (https://app.tracetest.io/organizations/tracetest-organization-id/environments/tracetest-environment-id/run/58WmDuBSR)
+ Summary: 1 passed, 0 failed, 0 pending
+ β Pokeshop - List (https://app.tracetest.io/organizations/tracetest-organization-id/environments/tracetest-environment-id/test/R5NITR14g/run/1/test) - trace id: 3cdcb56d6c226f7083f45d6b3d278051
+ β span[tracetest.span.type="http"]
+ β span[tracetest.span.type="database"]
```
+
+You can click the run link to open the test run in the Tracetest app. From there you can explore the generated trace, update the assertions and get the test definition to execute your test once more.
+
+
+
+You can also explore the test execution from the Testkube dashboard.
+
+
+
+## What's Next?
+
+After running this quickstart, you can now add Tracetest to the native CI/CD pipeline in your Kubernetes cluster. It allows you to execute scheduled test runs and synthetic tests. All while following the trace-based testing principle and enabling full in-depth assertions against trace data, not just the response.
+
+## Learn More
+
+Please visit our [documentation](https://docs.tracetest.io/) and join our [Slack Community](https://dub.sh/tracetest-community) for more info!
diff --git a/docs/docs/web-ui/creating-tests-traceid.mdx b/docs/docs/web-ui/creating-tests-traceid.mdx
index c421084532..acfdf53419 100644
--- a/docs/docs/web-ui/creating-tests-traceid.mdx
+++ b/docs/docs/web-ui/creating-tests-traceid.mdx
@@ -37,7 +37,7 @@ In this example, a TraceID Request has been chosen.
Select an existing variable set or create a new one in the drop down:
-
+
diff --git a/docs/sidebars.js b/docs/sidebars.js
index c38f50d5bc..e27fcf01bc 100644
--- a/docs/sidebars.js
+++ b/docs/sidebars.js
@@ -744,29 +744,29 @@ const sidebars = {
},
{
type: "doc",
- id: "concepts/assertions",
- label: "Assertions",
+ id: "concepts/tests",
+ label: "Tests",
},
- // {
- // type: "doc",
- // id: "concepts/data-stores",
- // label: "Data Stores",
- // },
{
type: "doc",
- id: "concepts/variable-sets",
- label: "Variable Sets",
+ id: "concepts/test-suites",
+ label: "Test Suites",
+ },
+ {
+ type: "doc",
+ id: "concepts/runs",
+ label: "Runs",
+ },
+ {
+ type: "doc",
+ id: "concepts/run-groups",
+ label: "Run Groups",
},
{
type: "doc",
id: "concepts/selectors",
label: "Selectors",
},
- // {
- // type: "doc",
- // id: "concepts/tests",
- // label: "Tests",
- // },
{
type: "doc",
id: "concepts/expressions",
@@ -774,14 +774,24 @@ const sidebars = {
},
{
type: "doc",
- id: "concepts/test-suites",
- label: "Test Suites",
+ id: "concepts/assertions",
+ label: "Assertions",
+ },
+ {
+ type: "doc",
+ id: "concepts/variable-sets",
+ label: "Variable Sets",
},
{
type: "doc",
id: "concepts/ad-hoc-testing",
label: "Ad-hoc Testing",
},
+ // {
+ // type: "doc",
+ // id: "concepts/data-stores",
+ // label: "Data Stores",
+ // },
{
type: "doc",
id: "concepts/versioning",