Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Only run E2E test apps that are affected on PRs #14254

Draft
wants to merge 16 commits into
base: develop
Choose a base branch
from

Conversation

mydea
Copy link
Member

@mydea mydea commented Nov 13, 2024

This PR aims to solve two things:

  1. Do not require us to keep a list of test-applications to run on CI. This is prone to be forgotten, and then certain tests do not run on CI and we may not notice (this has happened more than once in the past 😬 )
  2. Do not run E2E test applications that are unrelated to anything that was changed.

With this PR, instead of keeping a list of E2E test jobs we want to run manually in the build.yml file, this is now inferred (on CI) by a script based on the nx dependency graph and some config in the test apps package.json. This is how it works:

  1. Pick all folders in test-applications, by default all of them should run. --> This will "fix" the problem we had sometimes that we forgot to add test apps to the build.yml so they don't actually run on CI 😬
  2. For each test app, look at it's dependencies (and devDependencies), and see if any of them has changed in the current PR (based on nx affected projects). So e.g. if you change something in browser, only E2E test apps that have any dependency that uses browser will run, so e.g. node-express will be skipped.
  3. Additionally, you can configure variants in the test apps package.json - instead of defining this in the workflow file, it has to be defined in the test app itself now.
  4. Finally, a test app can be marked as optional in the package.json, and can also have optional variants to run.
  5. The E2E test builds the required & optional matrix on the fly based on these things.
  6. In pushes (=on develop/release branches) we just run everything.
  7. If anything inside of the e2e-tests package changed, run everything. --> This could be further optimized to only run changed test apps, but this was the easy solution for now.

Copy link
Contributor

github-actions bot commented Nov 13, 2024

size-limit report 📦

Path Size % Change Change
@sentry/browser 22.77 KB - -
@sentry/browser - with treeshaking flags 21.53 KB - -
@sentry/browser (incl. Tracing) 35.26 KB - -
@sentry/browser (incl. Tracing, Replay) 71.98 KB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 62.35 KB - -
@sentry/browser (incl. Tracing, Replay with Canvas) 76.31 KB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 89.14 KB - -
@sentry/browser (incl. Feedback) 39.93 KB - -
@sentry/browser (incl. sendFeedback) 27.42 KB - -
@sentry/browser (incl. FeedbackAsync) 32.23 KB - -
@sentry/react 25.52 KB - -
@sentry/react (incl. Tracing) 38.22 KB - -
@sentry/vue 26.91 KB - -
@sentry/vue (incl. Tracing) 37.09 KB - -
@sentry/svelte 22.91 KB - -
CDN Bundle 24.13 KB - -
CDN Bundle (incl. Tracing) 37.04 KB - -
CDN Bundle (incl. Tracing, Replay) 71.7 KB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 77.05 KB - -
CDN Bundle - uncompressed 70.73 KB - -
CDN Bundle (incl. Tracing) - uncompressed 109.9 KB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 222.42 KB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 235.64 KB - -
@sentry/nextjs (client) 38.33 KB - -
@sentry/sveltekit (client) 35.84 KB - -
@sentry/node 134.28 KB -0.01% -1 B 🔽
@sentry/node - without tracing 96.46 KB - -
@sentry/aws-serverless 106.72 KB - -

View base workflow run

@mydea mydea force-pushed the fn/only-run-changed-e2e-tests branch from ae3d63c to fcd6b78 Compare November 13, 2024 15:19
@mydea mydea self-assigned this Nov 13, 2024
@mydea mydea force-pushed the fn/only-run-changed-e2e-tests branch from b87ab12 to 4960c3d Compare November 14, 2024 10:06
@mydea mydea force-pushed the fn/only-run-changed-e2e-tests branch from 924b4ac to 927af9e Compare November 14, 2024 11:00
@mydea mydea force-pushed the fn/only-run-changed-e2e-tests branch from 927af9e to c7df3bf Compare November 14, 2024 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant