Add documentation

.github/workflows/markdown-linter.yml

@@ -0,0 +1,10 @@
+name: Check Markdown links
+
+on: push
+
+jobs:
+ markdown-link-check:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@master
+ - uses: gaurav-nelson/github-action-markdown-link-check@v1

README.md

@@ -1 +1,61 @@
-# wasi-testsuite
+# WASI tests
+
+This repository contains WebAssembly System Interface (WASI) tests and a test executor for running WASI tests on a selected WebAssembly runtime.
+
+The repository only holds tests that have been included in the WASI, and it does not include tests for in-progress proposals or other experimental APIs. Test executor included in the repository can however be used to run tests defined for proposals along with tests defined in this repository.
+
+## Getting started
+
+1. Clone repository
+Use `prod/testsuite-base` branch as it already includes precompiled test binaries (see [Branch structure](#branch-structure) paragraph).
+```bash
+git clone --branch prod/testsuite-base git@github.com:WebAssembly/wasi-testsuite.git
+```
+2. Make sure there's already an adapter for the runtime in the [`adapters`](adapters) directory; if not, create one (see [the doc](doc/adapters.md) for details).
+3. Install python3
+ 1. Ubuntu
+ ```
+ $ sudo apt install python3 python3-pip
+ ```
+4. Install test runner dependencies:
+```bash
+python3 -m pip install -r test-runner/requirements.txt
+```
+5. Execute test suites from this repository
+```bash
+python3 test-runner/wasi_test_runner.py \
+ -t ./tests/assemblyscript/testsuite/ `# path to folders containing .wasm test files` \
+ ./tests/c/testsuite/ \
+ -r adapters/wasmtime.sh # path to a runtime adapter
+```
+
+## Contributing
+All contributions are very welcome. Contributors can help with:
+
+* adding or updating test cases,
+* improving test execution and reporting,
+* integrating with new WASM runtimes,
+
+and many others. We appreciate both code contributions as well as suggestions and bug reports.
+
+## Developer guide
+Here is some additional information for developers who are willing to contribute.
+
+### Directory structure
+* [`test-runner`](test-runner) - test executor scripts.
+* [`tests`](tests) - source code of WASI tests and build scripts. The folder contains subfolders for all supported languages.
+* [`.github`](.github) - CI workflow definitions.
+* [`doc`](doc) - additional documentation.
+
+### Programming languages for tests
+The repository currently consists of tests implemented in the following languages:
+* `C` (with [`wasi-libc`](https://github.com/WebAssembly/wasi-libc))
+* `AssemblyScript`.
+
+The list of supported languages can be extended if needed.
+
+### Branch structure
+Apart from development branches for various features, we identify the following branches as critical (i.e. they won't be removed or force-updated):
+
+* `main` - main branch of the repository. Use this branch for development (e.g. updating test cases, modifying test runner)
+* `prod/testsuite-base` - the branch is an up-to-date fork of the `main` branch but it also includes precompiled binaries. Use this branch for simply running tests and validating WASM runtimes (see [doc](doc/precompiled-binaries.md) for details).

doc/adapters.md

@@ -0,0 +1,63 @@
+# WASM Runtime Adapters
+
+The test runner is designed to support different types of WebAssembly runtimes. Because the command line interface for WASM runtime is not standardized, every runtime can arbitrarily define how parameters should be passed to the runtime.
+
+WASI test runner is designed to support as many WASM runtimes as possible, therefore runtime peculiarities aren't hardcoded in the runner.
+
+In order to integrate WASM runtime with a test runner, the user has to provide a `runtime adapter`. It's an executable file that takes command-line arguments and translates them to a runtime call.
+
+## Interface
+The adapter executable must accept the following command line parameters and execute actions associated with those parameters:
+* `--version` - prints to standard output name and version of the runtime in the following format: `<NAME> <VERSION>`
+* `--env <NAME=VAL>` - passes environment variable to WASM module. The parameter can be used multiple times in one call.
+* `--arg <ARG>` - passes argument `<ARG>` to WASM module. The parameter can be used multiple times in one call.
+* `--test-file <PATH>` - runs WASM module located in `<PATH>`
+
+The adapter must return exit code to the environment that was passed as an argument to the `proc_exit` function in WASM code. This can be verified by running the following code:
+
+```wat
+(module
+ (import "wasi_snapshot_preview1" "proc_exit" (func $fimport$0 (param i32)))
+ (memory $0 0)
+ (export "memory" (memory $0))
+ (export "_start" (func $0))
+ (func $0
+ (call $fimport$0
+ (i32.const 13)
+ )
+ )
+)
+```
+and check if the exit code is equal to `13`. There are also two test cases in Assembly Script test suite that verify the behavior:
+* [proc_exit-failure](../tests/assemblyscript/testsuite/proc_exit-failure.ts)
+* [proc_exit-success](../tests/assemblyscript/testsuite/proc_exit-success.ts)
+### Examples:
+
+Print runtime version:
+
+```bash
+$ ./adapter.sh --version
+wasmtime-cli 1.0.1
+```
+
+Run WASM module:
+
+```bash
+$ ./adapter.sh --arg a1 --arg a2 --env E1=env1 --env E2=env2 --test-file test.wasm
+# Expected to start test.wasm module with E1=env1 and E2=env2
+# environment variables defined and arguments a1, a2 passed to
+# the module.
+
+$ echo $?
+# should display value passed to proc_exit function in WASM code
+```
+
+## Examples
+
+See the [`adapters`](../adapters) directory for example adapters.
+
+## Contributions
+
+We prefer runtime maintainers to maintain adapters in their own repository We'll only maintain adapters for [Bytecode Alliance](https://bytecodealliance.org/) projects and we'll aim for compatibility with the most recent stable version.
+
+We'll accept pull requests for new adapters in this repository, but we can't guarantee we'll have the capacity to maintain them (so they might stop working with the new runtime release).

doc/precompiled-binaries.md

@@ -0,0 +1,6 @@
+# Accessing precompiled binaries for tests
+The main branch (`main`) of the repository contains test runner and source files for tests, but it does not include precompiled binaries that are used to run tests. Test binaries can be large and might affect developers who are willing to contribute to this repository, but don't have a fast internet connection.
+
+Not providing test binaries however affects users who want just to run tests (e.g. on CI). The repository contains build scripts to compile all the tests, but compilation requires installing dependencies (tests are written in multiple languages, and each of them needs a different set of dependencies). Installing dependencies might not be straightforward on all the systems, so we need to provide users with precompiled binaries.
+
+To maintain both a positive developer experience and smooth onboarding for users who want to test their runtime, we created a `prod/testsuite-base` branch that is a fork of the `main` branch, but it also includes all the precompiled tests. The binaries are located in the test source directories (`tests/*/testsuite/*.wasm`). The branch is updated for every change in the `main` branch (see `upload_test_binaries` in the [`compile-tests`](../.github/workflows/compile-tests.yml) workflow).