Add docker compose based testing

Our release process documentation recommends getting passing tox
results on as many popular linux distributions as time allows. Doing
this by hand is cumbersome, redundant, and error prone.

Add a directory with a helper script that automates the entire process
using docker compose and document its use in MAINTAINERS.rst.

Signed-off-by: Martí Bolívar <mbolivar@amperecomputing.com>

.gitignore

@@ -11,3 +11,4 @@ shippable/
 # htmlcov is generated by tox due to coverage gathering in pytest
 htmlcov/
 .dir-locals.el
+docker-testing/outdir

MAINTAINERS.rst

@@ -11,20 +11,34 @@ Pre-release test plan
 
      git checkout vX.Y-branch
 
-1. Make tox happy on the following first-party platforms:
+1. Make tox happy on the following first-party non-Linux platforms:
 
    - Windows 10
    - the latest macOS
-   - the latest Ubuntu LTS
 
-2. Make tox happy on other popular Linux distributions as resources allow.
-   Doing this in a container is fine.
+   Do this by hand and check for any anomalous warnings in the output.
+   Do not just trust CI.
+
+2. Make tox happy on other popular Linux distributions:
 
    - Arch
-   - the latest Ubuntu release (if different than the latest LTS)
-   - Debian stable (if its Python 3 is still supported)
+   - the latest Ubuntu LTS (22.04 at time of writing)
+   - the latest Ubuntu release (23.04 at time of writing)
+   - Debian stable (12 at time of writing)
    - Debian testing
-   - Fedora
+   - the latest Fedora (38 at time of writing)
+
+   Automated infrastructure for doing this in docker is in the docker-testing
+   directory. Start by updating the Dockerfiles and compose.yaml in that
+   directory if any newer distribution versions should be tested.
+
+   Then, install docker compose in your host Linux environment and run::
+
+     cd docker-testing
+     ./run-tests.sh
+
+   Make sure to check the tox.log files mentioned in the output for any
+   anomalous warnings.
 
 3. Build alpha N (N=1 to start, then N=2 if you need more commits, etc.) and
    upload to pypi. See "Building and uploading the release wheels" below for

docker-testing/README.rst

@@ -0,0 +1,5 @@
+Docker based testing
+--------------------
+
+This directory contains helper files used for running west's tests in Docker on
+various Linux runtimes. It was originally developed for release testing.

docker-testing/arch/Dockerfile

@@ -0,0 +1,9 @@
+FROM archlinux:latest
+CMD ["/west/docker-testing/in-container-test.sh"]
+
+RUN pacman -Syu --noconfirm \
+           git \
+           python-pip \
+    && pacman -Scc --noconfirm
+
+RUN pip3 install --break-system-packages tox

docker-testing/compose.yaml

@@ -0,0 +1,76 @@
+# It would be nicer to consolidate the common user/volumes
+# boilerplate, but there wasn't enough time to figure out if it was
+# possible at time of writing. Improvements welcome.
+
+services:
+  west-arch:
+    build: arch
+    environment:
+      WEST_TOX_OUT: /west/docker-testing/outdir/arch
+      WEST_TOX_OUT_IN_HOST: ${WEST_IN_HOST}/docker-testing/outdir/arch
+
+    user: ${MY_UID}:${MY_GID}
+    volumes:
+      - /etc/passwd:/etc/passwd:ro
+      - /etc/group:/etc/group:ro
+      - ..:/west
+
+  west-debian-12:
+    build: debian-12
+    environment:
+      WEST_TOX_OUT: /west/docker-testing/outdir/debian-12
+      WEST_TOX_OUT_IN_HOST: ${WEST_IN_HOST}/docker-testing/outdir/debian-12
+
+    user: ${MY_UID}:${MY_GID}
+    volumes:
+      - /etc/passwd:/etc/passwd:ro
+      - /etc/group:/etc/group:ro
+      - ..:/west
+
+  west-debian-testing:
+    build: debian-testing
+    environment:
+      WEST_TOX_OUT: /west/docker-testing/outdir/debian-testing
+      WEST_TOX_OUT_IN_HOST: ${WEST_IN_HOST}/docker-testing/outdir/debian-testing
+
+    user: ${MY_UID}:${MY_GID}
+    volumes:
+      - /etc/passwd:/etc/passwd:ro
+      - /etc/group:/etc/group:ro
+      - ..:/west
+
+  west-fedora-38:
+    build: fedora-38
+    environment:
+      WEST_TOX_OUT: /west/docker-testing/outdir/fedora-38
+      WEST_TOX_OUT_IN_HOST: ${WEST_IN_HOST}/docker-testing/outdir/fedora-38
+
+    user: ${MY_UID}:${MY_GID}
+    volumes:
+      - /etc/passwd:/etc/passwd:ro
+      - /etc/group:/etc/group:ro
+      - ..:/west
+
+  west-ubuntu-22.04:
+    build: ubuntu-22.04
+    environment:
+      WEST_TOX_OUT: /west/docker-testing/outdir/ubuntu-22.04
+      WEST_TOX_OUT_IN_HOST: ${WEST_IN_HOST}/docker-testing/outdir/ubuntu-22.04
+
+    user: ${MY_UID}:${MY_GID}
+    volumes:
+      - /etc/passwd:/etc/passwd:ro
+      - /etc/group:/etc/group:ro
+      - ..:/west
+
+  west-ubuntu-23.04:
+    build: ubuntu-23.04
+    environment:
+      WEST_TOX_OUT: /west/docker-testing/outdir/ubuntu-23.04
+      WEST_TOX_OUT_IN_HOST: ${WEST_IN_HOST}/docker-testing/outdir/ubuntu-23.04
+
+    user: ${MY_UID}:${MY_GID}
+    volumes:
+      - /etc/passwd:/etc/passwd:ro
+      - /etc/group:/etc/group:ro
+      - ..:/west

docker-testing/debian-12/Dockerfile

@@ -0,0 +1,10 @@
+FROM debian:bookworm
+CMD ["/west/docker-testing/in-container-test.sh"]
+
+RUN apt-get update \
+    && apt-get install -y \
+        git \
+        python3-pip \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN pip3 install --break-system-packages tox

docker-testing/debian-testing/Dockerfile

@@ -0,0 +1,10 @@
+FROM debian:testing
+CMD ["/west/docker-testing/in-container-test.sh"]
+
+RUN apt-get update \
+    && apt-get install -y \
+        git \
+        python3-pip \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN pip3 install --break-system-packages tox

docker-testing/fedora-38/Dockerfile

@@ -0,0 +1,9 @@
+FROM fedora:38
+CMD ["/west/docker-testing/in-container-test.sh"]
+
+RUN dnf install -y \
+        git \
+        python3-pip \
+    && dnf clean dbcache
+
+RUN pip3 install tox

docker-testing/in-container-test.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+# This is the test script that runs in the containers themselves.
+
+WEST=/west
+
+die() {
+    [ ! -z "$@" ] && echo "error: $@" >&2
+    exit 1
+}
+
+# Verify the container environment set up meets this script's requirements.
+[ ! -z "$WEST_TOX_OUT" ] || die "missing $WEST_TOX_OUT"
+[ ! -z "$WEST_TOX_OUT_IN_HOST" ] || die "missing $WEST_TOX_OUT_IN_HOST"
+[ -d "$WEST" ] || die "missing $WEST in the container"
+
+TOX_LOG="$WEST_TOX_OUT/tox.log"
+TOX_LOG_IN_HOST="$WEST_TOX_OUT_IN_HOST/tox.log"
+WEST_TESTDIR="/tmp/west"
+
+mkdir "$WEST_TOX_OUT" || die "failed to make $WEST_TOX_OUT in container ($WEST_TOX_OUT_IN_HOST in host)"
+
+git clone -q "$WEST" "$WEST_TESTDIR" || die "failed to clone west to $WEST_TESTDIR in container"
+cd "$WEST_TESTDIR"
+
+echo "running tox, output in $TOX_LOG_IN_HOST in host"
+tox run >"$TOX_LOG" 2>&1 || die "tox failed"
+
+cp -R htmlcov "$WEST_TOX_OUT" || die "failed to copy coverage to $WEST_TOX_OUT_IN_HOST/htmlcov in host"

docker-testing/run-tests.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+# This is the top-level test script that runs in the host.
+
+HERE=$(dirname "$0")
+
+[ -d "$HERE/outdir" ] && rm -r "$HERE/outdir"
+
+set -e
+mkdir "$HERE/outdir"
+export MY_UID=$(id -u)
+export MY_GID=$(id -g)
+export WEST_IN_HOST=$(realpath "$HERE/..")
+docker-compose up --force-recreate --build

docker-testing/ubuntu-22.04/Dockerfile

@@ -0,0 +1,10 @@
+FROM ubuntu:22.04
+CMD ["/west/docker-testing/in-container-test.sh"]
+
+RUN apt-get update \
+    && apt-get install -y \
+        git \
+        python3-pip \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN pip3 install tox

docker-testing/ubuntu-23.04/Dockerfile

@@ -0,0 +1,10 @@
+FROM ubuntu:23.04
+CMD ["/west/docker-testing/in-container-test.sh"]
+
+RUN apt-get update \
+    && apt-get install -y \
+        git \
+        python3-pip \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN pip3 install --break-system-packages tox