You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is the entire checklist on what we should do to assert that Tracetest is working fine on each version release. On each version release, we can copy the contents of this checklist and open a Github Discussion to start each test.
On Tracetest, we work in two ways to test the entire system and guarantee that everything is working fine. One is through automatic tests, where we run unit, integration and end-to-end tests on Tracetest CLI, Web UI and Server.
Another source of tests is the manual tests that we execute on each release, simulating users, and checking if everything is ok on these tests.
Manual Tests
On our manual tests, we aim to do some sanity checks to see if the main features are working as expected. Usually, we run it on each Tracetest release.
Tests to validate Release Candidate
Test server installation via CLI
Docker Compose and no demo API
Docker Compose and demo API
Kubernetes and no demo API
Kubernetes and demo API
Test DataStore provisioning
Open WebUI and go to /settings page. The provisioned Data Store should be selected
Create a test on WebUI that calls a demo API (like Pokeshop or Open Telemetry Store). This test should fetch traces correctly and run without errors
Test Tracetest examples (we decided to skip DataStores testing for this release)
Test specific features added/changed on this release:
Create a test with a failed trigger using the WebUI
Create a test with a failed trace using the WebUI
Create a successful test with test specs using the WebUI
Create a test with a failed trigger using the CLI
Create a test with a failed trace using the CLI
Create a successful test with test specs using the CLI
Tests to validate Final Release
Pipeline
Check if our release pipeline on Release Tracetest workflow on Github Actions worked correctly
Test CLI updates
MacOS via homebrew
MacOS via curl script
Windows via chocolatey
Windows via manual download
Docs
Double check Detailed installation docs and see if everything is documented correctly
Test specific features added/changed on this release:
Create a test with a failed trigger using the WebUI
Create a test with a failed trace using the WebUI
Create a successful test with test specs using the WebUI
Create a test with a failed trigger using the CLI
Create a test with a failed trace using the CLI
Create a successful test with test specs using the CLI
Automatic Tests
Today Tracetest has 3 main components: a WebUI, a CLI and a Server.
Web UI
Unit tests: Run by executing npm test on ./web folder
End-to-end tests: Run using cypress against a temporary Tracetest created on Kubernetes.
CLI
Unit tests: Run by executing make test on ./cli folder
Server
Unit tests: Run by executing make test on ./server folder
Integration tests: Run along with some unit tests running make test on ./server folder, but also done in a matrix test on Github actions, by executing the test-examples action.
End-to-end tests: Run using Tracetest to test itself. We deploy two instances of Tracetest and use one to test API calls to another on Github actions, by executing the trace-testing action.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Testing Tracetest
This is the entire checklist on what we should do to assert that Tracetest is working fine on each version release. On each version release, we can copy the contents of this checklist and open a Github Discussion to start each test.
On Tracetest, we work in two ways to test the entire system and guarantee that everything is working fine. One is through automatic tests, where we run unit, integration and end-to-end tests on Tracetest CLI, Web UI and Server.
Another source of tests is the manual tests that we execute on each release, simulating users, and checking if everything is ok on these tests.
Manual Tests
On our manual tests, we aim to do some sanity checks to see if the main features are working as expected. Usually, we run it on each Tracetest release.
Tests to validate Release Candidate
Test server installation via CLI
Test DataStore provisioning
/settings
page. The provisioned Data Store should be selectedTest Tracetest examples (we decided to skip DataStores testing for this release)
Amazon X-Ray exampleDatadog exampleElastic APM exampleLightstep exampleNew Relic exampleSignalFX exampleTest specific features added/changed on this release:
Tests to validate Final Release
Pipeline
Test CLI updates
Docs
Test specific features added/changed on this release:
Automatic Tests
Today Tracetest has 3 main components: a WebUI, a CLI and a Server.
Web UI
npm test
on./web
folderCLI
make test
on./cli
folderServer
make test
on./server
foldermake test
on./server
folder, but also done in a matrix test on Github actions, by executing thetest-examples
action.trace-testing
action.Beta Was this translation helpful? Give feedback.
All reactions