doc: document how a test executor works

README.md

@@ -1,43 +1,55 @@
-# WASI tests
+# `wasi-testsuite`
 
-This repository contains tests for WebAssembly System Interface (WASI) and a test executor for running WASI tests on a selected WebAssembly runtime.
+This repository contains tests for WebAssembly System Interface (WASI) and a test executor for running WASI tests against a selected WebAssembly runtime.
 
-WASI is a modular collection of standardized APIs. Currently, WASI has not reached a v1 with a defined set of APIs.
-However, a snapshot of experimental APIs exists ([`wasi_snapshot_preview1`](https://github.com/WebAssembly/WASI/blob/main/legacy/preview1/docs.md)).
-The repository only holds tests of APIs included in this snapshot. It does not include tests for other in-progress proposals or other experimental APIs.
+WASI is a modular collection of standardized APIs. Currently, WASI has not reached version 1.0
+stability but a snapshot of experimental APIs does exist ([`wasi_snapshot_preview1`]). This
+repository only holds tests of APIs included in this snapshot. It does not include tests for other
+in-progress proposals or other experimental APIs, though the test executor can run tests from other repositories (e.g., see the [wasi-threads] tests).
 
-The test executor included in the repository can however be used to run tests defined for proposals along with tests defined in this repository.
+[`wasi_snapshot_preview1`]: https://github.com/WebAssembly/WASI/blob/main/legacy/preview1/docs.md
+[wasi-threads]: https://github.com/WebAssembly/wasi-threads/tree/main/test
 
-## Getting started
+The test executor matches execution output against a JSON specification. Writing your own test
+executor is quite simple; see the [specification] document for the details and the reference Python
+[implementation].
 
-1. Clone repository
- Use `prod/testsuite-base` branch as it already includes precompiled test binaries (see [Branch structure](#branch-structure) paragraph).
+[specification]: doc/specification.md
+[implementation]: ./test-runner
 
-```bash
-git clone --branch prod/testsuite-base git@github.com:WebAssembly/wasi-testsuite.git
-```
+## Getting started
+
+1. Clone the repository; use the `prod/testsuite-base` branch as it already includes precompiled
+ test binaries (see [branch structure](#branch-structure)):
 
-2. Make sure there's already an adapter for the runtime in the [`adapters`](adapters) directory; if not, create one (see [the doc](doc/adapters.md) for details).
-3. Install python3
- 1. Ubuntu
+ ```bash
+ git clone --branch prod/testsuite-base https://github.com/WebAssembly/wasi-testsuite
  ```
- $ sudo apt install python3 python3-pip
+
+2. Make sure there is already an adapter for the runtime in the [`adapters`](adapters) directory; if
+ not, create one (see [the doc](doc/adapters.md) for details).
+
+3. Install `python3` (e.g., on Ubuntu):
+
+ ```bash
+ sudo apt install python3 python3-pip
  ```
-4. Install test runner dependencies:
 
-```bash
-python3 -m pip install -r test-runner/requirements.txt
-```
+4. Install the test runner dependencies:
 
-5. Execute test suites from this repository
+ ```bash
+ python3 -m pip install -r test-runner/requirements.txt
+ ```
 
-```bash
-python3 test-runner/wasi_test_runner.py \
- -t ./tests/assemblyscript/testsuite/ `# path to folders containing .wasm test files` \
- ./tests/c/testsuite/ \
- ./tests/rust/testsuite/ \
- -r adapters/wasmtime.py # path to a runtime adapter
-```
+5. Execute the test suites from this repository:
+
+ ```bash
+ python3 test-runner/wasi_test_runner.py \
+ -t ./tests/assemblyscript/testsuite/ `# path to folders containing .wasm test files` \
+ ./tests/c/testsuite/ \
+ ./tests/rust/testsuite/ \
+ -r adapters/wasmtime.py # path to a runtime adapter
+ ```
 
 Optionally you can specify test cases to skip with the `--exclude-filter` option.
 
@@ -83,7 +95,11 @@ Here is some additional information for developers who are willing to contribute
 
 ### Cleaning up temporary resources
 
-Some of the tests (e.g. [pwrite-with-access](./tests/c/testsuite/pwrite-with-access.c)) generate output artifacts and their existence can affect consecutive test executions. Tests should clean up the artifacts they generate, but there might be cases where the test fails early. Test runner will automatically delete all the files and directories in the test suite directory with the `.cleanup` suffix.
+Some of the tests (e.g. [pwrite-with-access](./tests/c/testsuite/pwrite-with-access.c)) generate
+output artifacts and their existence can affect consecutive test executions. Tests should clean up
+the artifacts they generate, but there might be cases where the test fails early. The test runner
+will automatically delete all the files and directories in the test suite directory with the
+`.cleanup` suffix.
 
 ### Programming languages for tests
 
@@ -95,8 +111,6 @@ The repository currently consists of tests implemented in the following language
 
 The list of supported languages can be extended if needed.
 
-### Test
-
 ### Branch structure
 
 Apart from development branches for various features, we identify the following branches as critical (i.e. they won't be removed or force-updated):

doc/specification.md

@@ -0,0 +1,54 @@
+# Test Specification
+
+This document describes how to use the JSON test specifications to write your own test executor for
+[`wasi-testsuite`](..). The test executor included in this project provides a completely-usable
+reference [implementation](../test-runner), but projects with other requirements may want to run the
+tests in their own way (e.g., no Python dependency). The JSON test specifications provide a simple
+way to verify that the tests indeed passed.
+
+### Configuration
+
+Before executing anything, a test executor is expected to:
+- find all `*.wasm` files in a given subdirectory — these are the _test cases_
+- find all `*.cleanup` files in a given subdirectory and remove them — these are test
+ artifacts that can be generated during testing
+- for each test case, look for a `.json` file in the same directory matching the base name (e.g.,
+ `foo.json` for `foo.wasm`) — parse this _specification_
+- if no `.json` file is present, use a default specification; a (conceptual) default specification
+ would look like:
+
+ ```json
+ {
+ "args": [],
+ "dirs": [],
+ "env": {},
+ "exit_code": 0,
+ "stderr": "",
+ "stdout": ""
+ }
+ ```
+
+- if the specification is missing fields, use default values
+
+### Execution
+
+To execute the tests, the test executor is expected to:
+- `env`: construct an environment from each key-value pair in `env`; the environment should only
+ contain these keys and otherwise should be empty (note that this environment is the WASI
+ environment, whether the engine inherits the shell environment or explicitly configures it via
+ `--env` parameters)
+- `dir`: add each path listed in `dir` as WASI preopen directories (some engines use a `--dir`
+ flag)
+- `args`: pass each argument in order to the WASI program (most CLI engines allow appending these
+ after the module path)
+
+The test executor runs the WebAssembly test case with the above context and records its results.
+
+### Checking
+
+With the execution results in hand, the test executor is expected to:
+- `exit_code`: check that the WASI exit code matches `exit_code`, or `0` if none was provided
+- `stderr`: if a `stderr` field is provided, check that the bytes printed to `stderr` match it
+- `stdout`: if a `stdout` field is provided, check that the bytes printed to `stdout` match it
+
+A test case _passes_ if all of the checks are true.