YATM v2 (Yet Another Test Manager)

YATM v2 is a powerful tool for generating and managing test cases from requirements for manual testing. It's particularly useful for projects that rely on community contributors for testing, as it integrates seamlessly with GitHub issues.

YATM v2 is a rewrite of the original YATM tool, which was written in Typescript and used for several ROS 2 and Gazebo releases by Open Robotics and Intrinsic AI.

Key Features

Specify requirements in YAML files
Generate test cases from requirements using a customizable test case builder
CLI interface for easy management
Upload test cases to GitHub issues (with duplicate prevention)
Preview test cases locally in markdown before uploading
Initialize new YATM v2 projects with sensible defaults
Migrate requirements from YATM v1 to v2
Validate requirements and test case builder files
Get metrics on test cases from GitHub
Extensible design for custom requirement builders
Customizable GitHub issue template

Getting Started

Prerequisites

Install Rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Create a GitHub repository for your project.
Generate a GitHub personal access token with the following permissions:
- Actions (may be required)
- Contents
- Issues
- Pull requests

Installation

Clone the repository and build the project:

git clone https://github.com/paudrow/yatm-v2.git
cd yatm-v2
cargo build

Run the tests:
```
cargo test
```
Install YATM v2:
```
cargo install --path src/yatm_v2
```

Setting Up Your Workspace

Initialize a new YATM v2 workspace:
```
yatm_v2 init --path /path/to/your/workspace
```
This will create a new YATM v2 workspace with a config.yaml file and a requirements directory.
In config.yaml, modify the repo_owner and repo_name to match your Github repository.
Make sure that your workspace has access to your personal access token through the GITHUB_TOKEN environment variable.

YATM v2 will look in the .env file in the root of your workspace for the GITHUB_TOKEN environment variable. This file should look like this:
```
# .env
GITHUB_TOKEN="github_pat_**********************************************************************************"
```
You shouldn't commit this file to a public repository. It's already in the .gitignore file generated with each YATM v2 workspace.
Generate a preview of your test cases:
```
yatm_v2 github preview
```
This will generate a markdown file with your test cases in the generated_files (unless you've set a different path in config.yaml).

If this looks good, you can upload your test cases to Github.
Upload your test cases to Github:
```
yatm_v2 github upload
```
This will create a new issue on your Github repository with the test cases.

If you've gotten this far, you've successfully set up YATM v2 for your project. You can now start adding requirements and generating test cases.

Usage

Terminology

Requirement: A requirement specifies how a system should behave in a given situation. These are in the form of "Actions" that should be taken and "Expected Results" that should occur.
Test case: A test case is a requirement that has been given a specific testing environment, such as on Ubuntu 22.04 and with Cyclone DDS.

You can think of a requirement as an abstract class and a test case as an instance of that class.

Test case builder: A test case builder is a mapping of requirements to test cases. It specifies which requirements should be included and which permutations of several possible environments should be used.

Workflow

Initialize a new YATM v2 workspace with yatm_v2 init --path /path/to/your/workspace.
Modify the config.yaml file to match your Github repository.
Add a .env file to the root of your workspace with your Github personal access token.
(Optional) Prepare your repository by resetting the labels.
```
# type 'yes' when asked to confirm
yatm_v2 github utils delete-all-labels
```
Then you can create new labels specificed in the config.yaml file. There are some default labels that are created with yatm_v2 init, but you modify them or add more.
```
yatm_v2 github utils create-labels
```
Add requirements to the requirements directory. Optionally, verify that they are valid with yatm_v2 requirements validate.

You can generate a starter requirements file with the following command:
```
yatm_v2 requirements new
```
Create a test case builder in the test_cases_builder directory. Optionally, verify that it is valid with yatm_v2 test-cases validate.

You can generate a starter test case builder file with the following command:
```
yatm_v2 test-cases new
```
(Optional) Generate a preview of your test cases with yatm_v2 github preview.
Upload your test cases to Github with yatm_v2 github upload.

Create a meta issue to direct your users and create a list of links to make it easier to find the test cases.

yatm_v2 github make-label-links

This will give you something like the following in your project's generated_files directory:

- [`Build type: Debian`, `Chip set: AMD64`, `DDS: FastDDS`, `OS: Ubuntu Jammy 22.04`](https://github.com/paudrow/test-yatm-v2/issues?q=is:issue+is:open+label:%22Build+type:+Debian%22+label:%22Chip+set:+AMD64%22+label:%22DDS:+FastDDS%22+label:%22OS:+Ubuntu+Jammy+22.04%22)
- [`Build type: Binary`, `Chip set: AMD64`, `DDS: FastDDS`, `OS: Ubuntu Jammy 22.04`](https://github.com/paudrow/test-yatm-v2/issues?q=is:issue+is:open+label:%22Build+type:+Binary%22+label:%22Chip+set:+AMD64%22+label:%22DDS:+FastDDS%22+label:%22OS:+Ubuntu+Jammy+22.04%22)

(Optional) Get metrics on your test cases.
```
yatm_v2 github metrics
```
Which will give you something like the following:
```
2/18 issues closed: 11.11%
```
You can also get metrics for a specific label.
```
yatm_v2 github metrics --label "who: community tested"
```
Which will give you something like the following:
```
2/3 issues closed: 66.67%
```
This can be a great way to see how well we are doing with testing and to measure the impact of community contributions.
If you find that you need to make changes to your requirements or test case builder, you can
1. Close all of the issues on Github with the current workspace version.
```
yatm_v2 github close-all-issues --label "version: <current version>"
```
1. Bump the version of your workspace in config.yaml.
2. Upload your new test cases to Github.
```
yatm_v2 github upload
```

Parts of a requirements file

Here is an example of a requirements file:

requirements:
- name: name
  description: description
  steps:
  - name: step name
    description: step description
    action:
    - !Describe action
    - !StdIn
      number: 1
      text: echo 'hi'
    - !Url
      name: Google
      url: www.google.com
    - !Image https://placekitten.com/200/300
    expect:
    - !Describe expect
    - !StdOut
      number: 1
      text: hi
    - !StdErr
      number: 1
      text: error
    - !Url
      name: Google
      url: www.google.com
    - !Image https://placekitten.com/200/300
  labels:
  - label
  links:
  - name: Google
    url: www.google.com

The main thing to understand is that a requirement has one or more steps. Each step has an action and an expect. The action is what the user should do and the expect is what the system should output.

There are several different types of actions and expects. For example, !StdIn is an action that specifies that the user should input something into the system. !StdOut is an expect that specifies that the system should output something.

You'll see all of them in the example above.

One thing to note is that !StdIn, !StdOut, and !StdErr all have a number field. This is the terminal number for the action or expect. This is useful for when you have multiple actions or expects in a single step.

Also note that !Image will only use an image that can be accessed through a URL. You can put images on Github Gists and use the raw URL to use them in your requirements.

Parts of a test case builder file

Here is an example of a test case builder file:

test_cases_builders:
- name: Demo test cases
  description: description
  set:
  - !Include
    all_labels:
    - label
    any_names:
    - name
    negate: false
  - !Exclude
    all_labels: null
    any_names:
    - Demo
    negate: false
  labels:
  - Demo
  permutations:
    Operating System:
    - Ubuntu 22.04
    - Windows 11
    - MacOS 12.0
    RMW:
    - CycloneDDS
    - FastDDS

The test case builder has two main parts:

set: This is a set of requirements that should be included in the test cases. In this case, we're including all requirements.
permutations: This is a mapping of labels to a list of possible values. In this case, we're specifying that we want to generate test cases for each permutation of the Operating System and RMW labels.

Understanding the `!Include` and `!Exclude` directives

From the above example, we have the following:

test_cases_builders:
  ...
  set:
  - !Include
    all_labels:
    - label
    any_names:
    - name
    negate: false
  - !Exclude
    all_labels: null
    any_names:
    - Demo
    negate: false
  ...

Both !Include and !Exclude have the following fields:

all_labels: A list of labels that all requirements must have to be included or excluded. If null, then this field is ignored.

A label will match if it is an exact match of a requirement label. If more than one label is specified, then a requirement must have all of the labels to match.
any_names: A list of names that any requirement must have to be be included or excluded. If null, then this field is ignored.

A name will match if it is a substring of the requirement name. For example, if any_names is ['demo', 'foo'], then a requirement with the name Demo 1 will match (demo matches). Note that this is case insensitive.

If more than one name is specified, then a requirement must have at least one of the names to match.
negate: A boolean that specifies if the all_labels and any_names fields should be negated. If true, then the all_labels and any_names fields are negated.

For example, if !Include has all_labels: [A, B] and any_names: [C, D], then a requirement must have labels A and B and at least one of the names C and D to be included.

If !Include has negate: true, then a requirement must not have labels A and B and at least one of the names C and D to be included.

It's important to note that !Include and !Exclude are applied in the order they are specified in the set field.This means that you can include all requirements and then exclude some of them.

This is a powerful way to build test cases from requirements.

Understanding Permutations

In the above example we have the following:

  #...
  permutations:
    Operating System:
    - Ubuntu 22.04
    - Windows 11
    - MacOS 12.0
    RMW:
    - CycloneDDS
    - FastDDS

This will generate six test cases, one for each permutation of the Operating System and RMW labels.

Ubuntu 22.04, CycloneDDS
Ubuntu 22.04, FastDDS
Windows 11, CycloneDDS
Windows 11, FastDDS
MacOS 12.0, CycloneDDS
MacOS 12.0, FastDDS

For every key (Operating System and RMW), there will be a test case for each value.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

YATM v2 (Yet Another Test Manager)

Table of Contents

Key Features

Getting Started

Prerequisites

Installation

Setting Up Your Workspace

Usage

Terminology

Workflow

Parts of a requirements file

Parts of a test case builder file

Understanding the `!Include` and `!Exclude` directives

Understanding Permutations

Files

README.md

Latest commit

History

README.md

File metadata and controls

YATM v2 (Yet Another Test Manager)

Table of Contents

Key Features

Getting Started

Prerequisites

Installation

Setting Up Your Workspace

Usage

Terminology

Workflow

Parts of a requirements file

Parts of a test case builder file

Understanding the !Include and !Exclude directives

Understanding Permutations

Understanding the `!Include` and `!Exclude` directives