Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docker compose based test environment for Linux #687

Conversation

mbolivar-ampere
Copy link
Collaborator

When preparing a west release, we like to make sure that it works on more Linux environments than whatever is currently tested in GitHub CI. Doing this by hand is a pain. Add some automation for making this more painless.

This makes it trivial to inspect, line by line, what got covered.
That is useful when evaluating coverage for a new feature.

Signed-off-by: Martí Bolívar <mbolivar@amperecomputing.com>
@marc-hb
Copy link
Collaborator

marc-hb commented Sep 27, 2023

Nice! No time to review at this moment but if you're looking for some Windows inspiration you can find some here:

https://github.com/thesofproject/sof/blob/main/.github/workflows/zephyr.yml

https://github.com/thesofproject/sof/actions/workflows/daily-tests.yml

Copy link
Collaborator

@marc-hb marc-hb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dunno much about Docker unfortunately...

@@ -0,0 +1,29 @@
#!/bin/bash

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
set -e

It does not catch everything but it catches a lot.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its omission is intentional in this file -- see error handling being done manually below.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set -e is not incompatible with manual error handling at all. It would be totally unusable otherwise. set -e catches many failures that the developer did not anticipate or was too lazy or too optimistic to handle manually - now and in the future.

The only problem with set -e is the false, 100% safety impression that it can give to some inexperienced people. For everything else it's fine.

thesofproject/sof-test#312

WEST=/west

die() {
[ ! -z "$@" ] && echo "error: $@" >&2
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer an empty error: . No one should die without a message.

I also prefer a die that is fully printf compatible:

die()
{
        >&2 printf '%s ERROR: ' "$0"
        # We want die() to be usable exactly like printf
        # shellcheck disable=SC2059
        >&2 printf "$@"
        exit 1
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an error path when there's no argument. I'd rather not use printf syntax, though.

Copy link
Collaborator

@marc-hb marc-hb Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot this: this script is more than big enough to deserve a main() function.

Very small change, all the benefits listed in:

@marc-hb
Copy link
Collaborator

marc-hb commented Oct 7, 2023

I dunno much about Docker unfortunately...

One thing we learned in SOF though; the Dockerfiles should be in a separate repo:

Except for "unit" test code which is very dependent on the version of the production code it's testing, you usually want the convenience to combine any version of test code with any version of production code. Typically: git bisect production code with the latest version of test code. Or here: git bisect with the latest docker image.

Now don't get me wrong: anyone can take 2 random git repos mostly unrelated to each other, merge them into one and start interleaving their unrelated development histories. It is totally possible and even not that uncommon (especially when you don't know about west ;-) But it's really not convenient when the two projects have totally unrelated life cycles and release cycles.

Another instance: every Dockerfile change should trigger some Github Action that rebuilds the container and runs the test suite again. But any Dockerfile CI takes a very long time so you don't want that to happen for mosts west PRs which do NOT change any Dockerfile. Congratulations, now you need a path-based filter on every Github Action to separate two different CIs that should have never been together in the first place. Possible but pretty inconvenient.

Our release process documentation recommends getting passing tox
results on as many popular linux distributions as time allows. Doing
this by hand is cumbersome, redundant, and error prone.

Add a directory with a helper script that automates the entire process
using docker compose and document its use in MAINTAINERS.rst.

Signed-off-by: Martí Bolívar <mbolivar@amperecomputing.com>
@mbolivar-ampere
Copy link
Collaborator Author

One thing we learned in SOF though; the Dockerfiles should be in a separate repo:

This is meant for manual testing on various distros at release time; can we keep things simple for now?

Copy link
Collaborator

@marc-hb marc-hb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still no time to look at the docker files yet sorry.

@@ -0,0 +1,29 @@
#!/bin/bash

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set -e is not incompatible with manual error handling at all. It would be totally unusable otherwise. set -e catches many failures that the developer did not anticipate or was too lazy or too optimistic to handle manually - now and in the future.

The only problem with set -e is the false, 100% safety impression that it can give to some inexperienced people. For everything else it's fine.

thesofproject/sof-test#312

WEST=/west

die() {
if [ ! -z "$@" ]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"$@" expands to multiple strings. That's not what you want here
http://mywiki.wooledge.org/BashPitfalls#for_arg_in_.24.2A
http://mywiki.wooledge.org/BashGuide/Parameters#Special_Parameters_and_Variables

Suggested change
if [ ! -z "$@" ]; then
if [ -n "$*" ]; then

Or even just:

Suggested change
if [ ! -z "$@" ]; then
if [ -n "$1" ]; then

Or if [ $# -eq 0 ]

Then for consistency:

echo "error: $*" >&2

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW you don't HAVE to printf 'error: %s' "$var", you can still do printf "error: $var" if you want. The well-tested die() function I recommended earlier supports either indifferently.

TOX_LOG_IN_HOST="$WEST_TOX_OUT_IN_HOST/tox.log"
WEST_TESTDIR="/tmp/west"

mkdir "$WEST_TOX_OUT" || die "failed to make $WEST_TOX_OUT in container ($WEST_TOX_OUT_IN_HOST in host)"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit

Suggested change
mkdir "$WEST_TOX_OUT" || die "failed to make $WEST_TOX_OUT in container ($WEST_TOX_OUT_IN_HOST in host)"
mkdir "$WEST_TOX_OUT" ||
die "failed to make $WEST_TOX_OUT in container ($WEST_TOX_OUT_IN_HOST in host)"

(No need for a backslash in this case)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How likely is this mkdir to fail really? Isn't set -e enough?

main()
{
# Verify the container environment set up meets this script's requirements.
[ ! -z "$WEST_TOX_OUT" ] || die "missing $WEST_TOX_OUT"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
[ ! -z "$WEST_TOX_OUT" ] || die "missing $WEST_TOX_OUT"
[ -n "$WEST_TOX_OUT" ] || die "missing $WEST_TOX_OUT"

The fewer negations the better and copy/paste of ! can trigger history recall in interactive testing on the command line.

@marc-hb
Copy link
Collaborator

marc-hb commented Sep 23, 2024

Resurrected by @pdgendt in newer #733

@marc-hb marc-hb closed this Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants