-
Notifications
You must be signed in to change notification settings - Fork 5
Running github workflows locally
It is possible to run github workflows locally using a tool called Act https://github.com/nektos/act
To do this you will need a local docker instance, and github cli
If you are on a Diamond workstation without root, you can install rootless docker: https://docs.docker.com/engine/security/rootless/ (Click the installation tab "without packages")
curl -fsSL https://get.docker.com/rootless > rootless
SKIP_IPTABLES=1 sh ./rootless
systemctl --user status docker
You may also need to edit the daemon config to put the data dir on the scratch folder:
vim ~/.config/docker/daemon.json
{
"data-root": "/scratch/ws/docker/docker-root"
}
https://cli.github.com/manual/
If you don't already have it, you can install prebuilt version with conda
https://confluence.diamond.ac.uk/display/SSCC/How+to+use+Conda+at+Diamond
module load mamba
mamba create --prefix /scratch/conda/conda-env
conda activate /scratch/conda/conda-env
conda install gh --channel conda-forge
Then install act into GH cli:
conda install gh --channel conda-forge
gh extension install https://github.com/nektos/gh-act
When running act, it will expect the current directory to be a checked-out repository, so it is best to run against a separate cloned repo so that your dev repo is the upstream. After that:
gh auth login
cd <path-to-hyperion-repo>
gh help act
For information on available workflow actions, consult github workflow documentation: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows
Changes are written to the overlay file system, to see the "live" working tree, you need to shell into the docker container:
docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
80af536078e4 catthehacker/ubuntu:act-latest "tail -f /dev/null" 21 minutes ago Up 21 minutes act-pre-release-workflow-pin-dependency-versions-c2252e748e6f7f7d9c295ad8eb1d5f2231befc8d042db1a7e4d623530c0f859b
docker container exec -t -i 80af536078e4 /bin/bash
If you want to test python workflow scripts inside of the container, the actions/setup-python action installs python in /opt/hostedtoolcache/Python
To perform github actions in the workflow pass your github token to act via the -s option, this will also get rid of rate-limit warnings when downloading.
In order to push commits to a local repo instead of github, the origin reference of the repo in the docker container will need to be accessible.
To do this, you can make ssh keys available to the docker container via a bind mount and set the identity used by the ssh command called by git:
gh act \
--container-options "-v /scratch/github-workflow-test/bind:/bind" \
--env-file ../secrets \
-s GITHUB_TOKEN=<my_github_token> \
-P ubuntu-latest=catthehacker/ubuntu:full-latest \
<workflow_name> [inputs...]
secrets contains an environment file with
GIT_SSH_COMMAND=ssh -i /bind/id_ed25519 -o UserKnownHostsFile=/bind/known_hosts
in the bind directory put ssh configuration files: populate with a .ssh private key id_ed25519 and add the origin server to the known_hosts
The default docker images used by act are small and don't contain all features. In order to run github cli in workflows (and possibly other things) you may need to specify a different docker image (see https://nektosact.com/usage/runners.html) using the -P option.
-P ubuntu-latest=catthehacker/ubuntu:full-latest
If running rootless docker, the full-latest images will run into permission issues accessing your SSH private key as above as unlike the default images the workflow runs as the runner user instead of root, to fix this you can create your own docker image with the key chowned to the correct user with a Dockerfile similar to
FROM catthehacker/ubuntu:full-latest
ADD --chown=runner:runner bind /bind
then build it with docker build .
you will also need to specify --pull=false to prevent act from trying to pull your image from dockerhub.