Skip to content

Latest commit

 

History

History

integration-tests

HAC UI Tests

Functional tests using Cypress

What can be found here

The important bits are as follows:

Path Description
cypress.config.ts cypress configuration file, contains all the environment and plugin setup
support contains helper files for cypress, like hooks, custom commands, page objects or plugin integration
tests the actual spec files reside here
utils utilities for easier test development, from UI interaction tasks to network requests
cypress test results, including screenshots, video recordings, HTML reports, xunit files, etc.

Running the tests locally

Prerequisites:

  • Nodejs (any version that is not pre-historical will do)
  • Access (credentials) to running RHTAP instance
  • Github token for the advanced-happy-path spec (can be obtained from openshift-ci vault for hac-dev, contact jrichter for access)

Certain environment variables will need to be set up, depending on what you intend to run. The most convenient way is to export them from the CLI, in which case they need to be prefixed with CYPRESS_, e.g. USERNAME is set as export CYPRESS_USERNAME=username. Alternatively, they can be passed to cypress via -e flag, e.g npx cypress run -e USERNAME=username.

Find the supported variables in the table below:

Variable name Description Required Default Value
HAC_BASE_URL The URL to the main page of RHTAP UI Yes 'https://prod.foo.redhat.com:1337/beta/hac/application-pipeline'
USERNAME Username for SSO login Yes ''
PASSWORD Password for SSO login Yes ''
PR_CHECK Assume the test is a PR check, enable report portal, assume keycloak is used instead of RH-SSO For PR checks false
PERIODIC_RUN Assume the test is a nightly run, keycloak is used instead of RH-SSO For runs using ephemeral RHTAP environment false
REMOVE_APP_ON_FAIL Clean up applications from the cluster even if tests fail No false
GH_TOKEN Github token for network requests For the advanced spec ''
RP_TOKEN Report portal token for PR checks No ''
GH_PR_TITLE Report portal setup No ''
GH_PR_LINK Report portal setup No ''

Running from source

This is the recommended way when either developing tests, or a headed test runner is preferred. After running npm install and setting the appropriate environment variables, use one of the following commands to run cypress.

For the headed test runner:

$ npx cypress open

Select E2E testing and the browser of your choice (chrome is recommended though), then launch any spec by clicking it in the list, and watch as it runs.

For headless execution:

$ npx cypress run

Runs all available specs, by default in the Electron browser. Flags that might come in handy are -b to specify the browser, or -s to filter spec files based on a glob pattern. For example, running the basic happy path spec in chrome:

$ npx cypress run -b chrome -s 'tests/basic-happy-path*'

Running using docker image

This folder contains a Dockerfile that specifies an image with all the dependencies and test code included. It should be available on quay.io and should be updated no more than 30 minutes after a change is pushed to this folder.

If there are no changes in your local test code, you can pull and run the image from quay, providing the required environment variables. Feel free to use docker or podman, we will be using podman in this example:

$ podman run -e CYPRESS_HAC_BASE_URL=https://<HOSTNAME>/hac/application-pipeline -e CYPRESS_USERNAME="user1" -e CYPRESS_PASSWORD="user1" quay.io/hacdev/hac-tests:next

Note that the container specific arguments like environment and mountpoints are defined before the image tag. Any arguments defined after the image tag will be interpreted as options and passed directly to cypress. In general, the the command structure is as follows:

$ <container engine> run <mount points> <container environment variables> quay.io/hacdev/hac-tests:next <cypress arguments>

Since the image already contains all the test code, in case you'd like to run the tests with your local changes, you would need to mount the local code:

$ podman run -v <path to integration-tests>:/e2e:Z -e CYPRESS_HAC_BASE_URL=https://<HOSTNAME>/hac/application-pipeline -e CYPRESS_USERNAME="user1" -e CYPRESS_PASSWORD="user1" quay.io/hacdev/hac-tests:next

The entrypoint searches for any code within /e2e path and runs it instead of any code that was already present inside the image.

Passing arguments to cypress

The cypress run command can be customized using flags, as per the documentation. To pass these into the entrypoint, simply add them to the end of your container run command.

For example, we can limit what spec files will be run:

$ podman run -e CYPRESS_HAC_BASE_URL=https://<HOSTNAME>/hac/application-pipeline quay.io/hacdev/hac-tests:next --spec tests/basic-happy-path.spec.ts

Or if instead of using container environment variables, you prefer to pass them directly to cypress:

$ podman run quay.io/hacdev/hac-tests:next -e HAC_BASE_URL=https://<HOSTNAME>/hac/application-pipeline

There is no need for the CYPRESS prefix this way. Passing HAC_BASE_URL is equivalent to setting a variable in the container called CYPRESS_HAC_BASE_URL.

Keep in mind that it is the image tag that divides the command between container setup on its left, and the entrypoint arguments on the right (those will be passed to cypress).

Accessing test results

Test artifacts (reports, screenshots, videos, etc.) are only accessible to the host system if the appropriate container folder is mounted.

When running with local test code mounted, with the -v <path to integration-tests>:/e2e:Z option, all the test artifacts are available to the host at <path to integration-tests>/cypress.

When running the container with the included test code, the tests artifacts are available inside the container at /tmp/artifacts. Mouning the folder, the host can then access the artifacts at the <chosen path>:

$ podman run -v <chosen path>:/tmp/artifacts:Z <environment variables> quay.io/hacdev/hac-tests:next

Building the image

To build the image locally, navigate to this folder and run (using your favorite container runtime)

$ podman build -f Dockerfile -t <awesome tag>

Publishing the image

The image is being published automatically after a change to this folder is pushed to main. If however, the need arises to publish it manually, you will first need access to the quay.io/hacdev/hac-tests repository, ask jrichter, kfoniok, or skhileri for rights.

Of course we will now need to build the image with the appropriate tag and push it (after logging in):

$ podman build -f Dockerfile -t quay.io/hacdev/hac-tests:next
$ podman login -u="$USERNAME" -p="$TOKEN" quay.io
$ podman push quay.io/hacdev/hac-tests:next

Tests on CI

For CI we are using a dual cluster setup. The frontend is deployed per run on an ephemeral cluster using bonfire, and generally only lives as long as the CI run is active. The backend RHTAP runs on an OSD cluster that we maintain and keep up to date with the latest RHTAP. For access to our CI cluster, contact kfoniok. The backend configuration can be found in our fork of infra-deployments.

The two clusters communicate through a proxy. The frontend SSO url is also replaced with a url to a keycloak instance running on the CI cluster. For each CI run, a new user is generated and registered in the toolchain host operator.

Finally, the tests are run using the quay.io/hacdev/hac-tests:next image.

For step-by-step setup, refer to the pr_check.sh file.

PR checks

Our Cypress tests run on every pull request, using a jenkins job (VPN required). The job runs the aforementioned pr_check.sh file. Note that even though the job exposes all results generated by the tests, the HTML report will not open properly, unless downloaded, due to javascript restrictions on the jenkins instance.

By default, the advanced-happy-path spec file (component with custom build pipeline) is skipped for PR checks, due to long run time (~20 minutes). To run the full suite of tests, comment [test] on the PR. To rerun the default suite, comment /retest.

The job also uploads results to the RHTAP QE report portal (ask jrichter for access if interested).

Periodic tests

Additionally, the tests also run nightly on openshift-ci. Unlike the PR check, this job always runs the entire test suite.

The job history can be found here, the job definition is available here.

The test results will also be reported on the #forum-rhtap-test-execution-alerts slack channel.

Infra-deployments PR checks

The upstream infra-deployments repo is also starting to incorporate our cypress tests in their PR checks to test new backend changes against our frontend. Currently, the basic-happy-path spec is being used.

Job history is available here.

Reporting issues

If you find a problem with the tests, feel free to open an issue at the HAC Jira project. Make sure the component field is set to developer and label it as qe.

If you discover a production bug thanks to a test failure, please label any issue created for that bug as ci-fail. That way we can see all this automation has actually generated some value.