Functional tests using Cypress
The important bits are as follows:
Path | Description |
---|---|
cypress.config.ts |
cypress configuration file, contains all the environment and plugin setup |
support |
contains helper files for cypress, like hooks, custom commands, page objects or plugin integration |
tests |
the actual spec files reside here |
utils |
utilities for easier test development, from UI interaction tasks to network requests |
cypress |
test results, including screenshots, video recordings, HTML reports, xunit files, etc. |
Prerequisites:
- Nodejs (any version that is not pre-historical will do)
- Access (credentials) to running RHTAP instance
- Github token for the
advanced-happy-path
spec (can be obtained from openshift-ci vault for hac-dev, contactjrichter
for access)
Certain environment variables will need to be set up, depending on what you intend to run. The most convenient way is to export them from the CLI, in which case they need to be prefixed with CYPRESS_
, e.g. USERNAME
is set as export CYPRESS_USERNAME=username
. Alternatively, they can be passed to cypress via -e
flag, e.g npx cypress run -e USERNAME=username
.
Find the supported variables in the table below:
Variable name | Description | Required | Default Value |
---|---|---|---|
HAC_BASE_URL |
The URL to the main page of RHTAP UI | Yes | 'https://prod.foo.redhat.com:1337/beta/hac/application-pipeline' |
USERNAME |
Username for SSO login | Yes | '' |
PASSWORD |
Password for SSO login | Yes | '' |
PR_CHECK |
Assume the test is a PR check, enable report portal, assume keycloak is used instead of RH-SSO | For PR checks | false |
PERIODIC_RUN |
Assume the test is a nightly run, keycloak is used instead of RH-SSO | For runs using ephemeral RHTAP environment | false |
REMOVE_APP_ON_FAIL |
Clean up applications from the cluster even if tests fail | No | false |
GH_TOKEN |
Github token for network requests | For the advanced spec |
'' |
RP_TOKEN |
Report portal token for PR checks | No | '' |
GH_PR_TITLE |
Report portal setup | No | '' |
GH_PR_LINK |
Report portal setup | No | '' |
This is the recommended way when either developing tests, or a headed test runner is preferred. After running npm install
and setting the appropriate environment variables, use one of the following commands to run cypress.
For the headed test runner:
$ npx cypress open
Select E2E testing and the browser of your choice (chrome is recommended though), then launch any spec by clicking it in the list, and watch as it runs.
For headless execution:
$ npx cypress run
Runs all available specs, by default in the Electron browser. Flags that might come in handy are -b
to specify the browser, or -s
to filter spec files based on a glob pattern. For example, running the basic happy path spec in chrome:
$ npx cypress run -b chrome -s 'tests/basic-happy-path*'
This folder contains a Dockerfile that specifies an image with all the dependencies and test code included. It should be available on quay.io and should be updated no more than 30 minutes after a change is pushed to this folder.
If there are no changes in your local test code, you can pull and run the image from quay, providing the required environment variables. Feel free to use docker or podman, we will be using podman in this example:
$ podman run -e CYPRESS_HAC_BASE_URL=https://<HOSTNAME>/hac/application-pipeline -e CYPRESS_USERNAME="user1" -e CYPRESS_PASSWORD="user1" quay.io/hacdev/hac-tests:next
Note that the container specific arguments like environment and mountpoints are defined before the image tag. Any arguments defined after the image tag will be interpreted as options and passed directly to cypress. In general, the the command structure is as follows:
$ <container engine> run <mount points> <container environment variables> quay.io/hacdev/hac-tests:next <cypress arguments>
Since the image already contains all the test code, in case you'd like to run the tests with your local changes, you would need to mount the local code:
$ podman run -v <path to integration-tests>:/e2e:Z -e CYPRESS_HAC_BASE_URL=https://<HOSTNAME>/hac/application-pipeline -e CYPRESS_USERNAME="user1" -e CYPRESS_PASSWORD="user1" quay.io/hacdev/hac-tests:next
The entrypoint searches for any code within /e2e
path and runs it instead of any code that was already present inside the image.
The cypress run command can be customized using flags, as per the documentation. To pass these into the entrypoint, simply add them to the end of your container run command.
For example, we can limit what spec files will be run:
$ podman run -e CYPRESS_HAC_BASE_URL=https://<HOSTNAME>/hac/application-pipeline quay.io/hacdev/hac-tests:next --spec tests/basic-happy-path.spec.ts
Or if instead of using container environment variables, you prefer to pass them directly to cypress:
$ podman run quay.io/hacdev/hac-tests:next -e HAC_BASE_URL=https://<HOSTNAME>/hac/application-pipeline
There is no need for the CYPRESS
prefix this way. Passing HAC_BASE_URL
is equivalent to setting a variable in the container called CYPRESS_HAC_BASE_URL
.
Keep in mind that it is the image tag that divides the command between container setup on its left, and the entrypoint arguments on the right (those will be passed to cypress).
Test artifacts (reports, screenshots, videos, etc.) are only accessible to the host system if the appropriate container folder is mounted.
When running with local test code mounted, with the -v <path to integration-tests>:/e2e:Z
option, all the test artifacts are available to the host at <path to integration-tests>/cypress
.
When running the container with the included test code, the tests artifacts are available inside the container at /tmp/artifacts
. Mouning the folder, the host can then access the artifacts at the <chosen path>
:
$ podman run -v <chosen path>:/tmp/artifacts:Z <environment variables> quay.io/hacdev/hac-tests:next
To build the image locally, navigate to this folder and run (using your favorite container runtime)
$ podman build -f Dockerfile -t <awesome tag>
The image is being published automatically after a change to this folder is pushed to main. If however, the need arises to publish it manually, you will first need access to the quay.io/hacdev/hac-tests
repository, ask jrichter
, kfoniok
, or skhileri
for rights.
Of course we will now need to build the image with the appropriate tag and push it (after logging in):
$ podman build -f Dockerfile -t quay.io/hacdev/hac-tests:next
$ podman login -u="$USERNAME" -p="$TOKEN" quay.io
$ podman push quay.io/hacdev/hac-tests:next
For CI we are using a dual cluster setup. The frontend is deployed per run on an ephemeral cluster using bonfire
, and generally only lives as long as the CI run is active. The backend RHTAP runs on an OSD cluster that we maintain and keep up to date with the latest RHTAP. For access to our CI cluster, contact kfoniok
. The backend configuration can be found in our fork of infra-deployments.
The two clusters communicate through a proxy. The frontend SSO url is also replaced with a url to a keycloak instance running on the CI cluster. For each CI run, a new user is generated and registered in the toolchain host operator.
Finally, the tests are run using the quay.io/hacdev/hac-tests:next
image.
For step-by-step setup, refer to the pr_check.sh file.
Our Cypress tests run on every pull request, using a jenkins job (VPN required). The job runs the aforementioned pr_check.sh
file. Note that even though the job exposes all results generated by the tests, the HTML report will not open properly, unless downloaded, due to javascript restrictions on the jenkins instance.
By default, the advanced-happy-path
spec file (component with custom build pipeline) is skipped for PR checks, due to long run time (~20 minutes). To run the full suite of tests, comment [test]
on the PR. To rerun the default suite, comment /retest
.
The job also uploads results to the RHTAP QE report portal (ask jrichter
for access if interested).
Additionally, the tests also run nightly on openshift-ci. Unlike the PR check, this job always runs the entire test suite.
The job history can be found here, the job definition is available here.
The test results will also be reported on the #forum-rhtap-test-execution-alerts
slack channel.
The upstream infra-deployments repo is also starting to incorporate our cypress tests in their PR checks to test new backend changes against our frontend. Currently, the basic-happy-path
spec is being used.
Job history is available here.
If you find a problem with the tests, feel free to open an issue at the HAC Jira project. Make sure the component field is set to developer
and label it as qe
.
If you discover a production bug thanks to a test failure, please label any issue created for that bug as ci-fail
. That way we can see all this automation has actually generated some value.