Add support for test filtering via tags

[yannbf]

Closes: #281, Closes: #218, Closes: #74, Closes: #306, Closes: #247

This PR introduces the concept of test filtering via the story tags annotation.

Filtering tests (experimental)

You might want to skip certain stories in the test-runner, run tests only against a subset of stories, or exclude certain stories entirely from your tests. This is possible via the tags annotation.

This annotation can be part of a story, therefore only applying to it, or the component meta (the default export), which applies to all stories in the file:

const meta = {
  component: Button,
  tags: ['design', 'test-only'],
};
export default meta;

// will inherit tags from meta with value ['design', 'test-only']
export const Primary = {};

export const Secondary = {
  // will override tags to be just ['skip']
  tags: ['skip'],
};

Note
You can't import constants from another file and use them to define tags in your stories. The tags in your stories or meta have to be defined inline, as an array of strings. This is a limitation due to Storybook's static analysis.

Once your stories have your own custom tags, you can filter them via the tags property in your test-runner configuration file. You can also use the CLI flags --includeTags, --excludeTags or --skipTags for the same purpose. The CLI flags will take precedence over the tags in the test-runner config, therefore overriding them.

Test-runner config: tags

The tags property contains three options: include | exclude | skip, each accepting an array of strings:

// .storybook/test-runner.js
module.exports = {
  tags: {
    include: [], // string array, e.g. ['test-only']
    exclude: [], // string array, e.g. ['design', 'docs-only']
    skip: [], // string array, e.g. ['design']
  },
};

📦 Published PR as canary version: 0.15.0--canary.382.449be19.0

✨ Test out this PR locally via:

npm install @storybook/test-runner@0.15.0--canary.382.449be19.0
# or 
yarn add @storybook/test-runner@0.15.0--canary.382.449be19.0

Version

Published prerelease version: v0.15.0-next.0

Changelog

🚀 Enhancement

• Add support for test filtering via tags #382 (@yannbf)

🐛 Bug Fix

• fix: switch checkStorybook to use GET method #385 (@stramel)
• Chore: update wait-on and lockfile #389 (@yannbf)
• Add storyshots migration guides #380 (@yannbf @shilman @jonniebigodes)
• Fix: Do not set test root outside index json mode #387 (@yannbf)

Authors: 4

• @jonniebigodes
• Michael Shilman (@shilman)
• Michael Stramel (@stramel)
• Yann Braga (@yannbf)

[codecov]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (5f9b1eb) 76.66% compared to head (449be19) 80.64%.

Files	Patch %	Lines
src/playwright/transformPlaywrightJson.ts	87.50%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             next     #382      +/-   ##
==========================================
+ Coverage   76.66%   80.64%   +3.97%     
==========================================
  Files          11       12       +1     
  Lines         180      217      +37     
  Branches       40       56      +16     
==========================================
+ Hits          138      175      +37     
  Misses         42       42              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shilman]

Looks great! Minor comment on the code. Also, you might want to provide a recipe for how to only execute tests with a play function using tags, since that's a frequently requested feature.

[github-actions]

🚀 PR was released in v0.15.0 🚀

[IanVS]

Not sure where the best place to ask this is, but what is the difference between skipTags and excludeTags? From reading the description, both of those will cause stories with those tags not to execute in the test runner, right? Why have two different options that do the same (?) thing?

Edit: I see, skipTags still shows the story in the cli output:

vs excludeTags doesn't:

Personally I don't see a compelling reason to support both options, and probably would have just started with excludeTags for now until/unless people asked for a way to also list them as skipped. But since both now exist, I'd suggest we need to clarify the difference between them (maybe "Mark stories as skipped" vs "Skip test stories").

[yannbf]

Not sure where the best place to ask this is, but what is the difference between skipTags and excludeTags? From reading the description, both of those will cause stories with those tags not to execute in the test runner, right? Why have two different options that do the same (?) thing?

Edit: I see, skipTags still shows the story in the cli output:

vs `excludeTags` doesn't: Personally I don't see a compelling reason to support both options, and probably would have just started with `excludeTags` for now until/unless people asked for a way to also list them as skipped. But since both now exist, I'd suggest we need to clarify the difference between them (maybe "Mark stories as skipped" vs "Skip test stories").

Thanks for your input Ian! And thanks for working on the clarification in the docs. Skipping is quite useful when there are temporary issues that you want to disable tests for, but you don't want to forget that they exist