From b8cb300fdc8ee2253f71d7ccad035bc82aa14f46 Mon Sep 17 00:00:00 2001 From: Declan Valters Date: Mon, 16 Sep 2024 10:33:42 +0100 Subject: [PATCH 1/6] TRE Container Examples --- .../container-examples.md | 26 +++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 27 insertions(+) create mode 100644 docs/safe-haven-services/tre-container-user-guide/container-examples.md diff --git a/docs/safe-haven-services/tre-container-user-guide/container-examples.md b/docs/safe-haven-services/tre-container-user-guide/container-examples.md new file mode 100644 index 000000000..c63a27826 --- /dev/null +++ b/docs/safe-haven-services/tre-container-user-guide/container-examples.md @@ -0,0 +1,26 @@ +# Container Examples + +To help with writing your own Dockerfiles to run within a Trusted Research Environment via the Container Execution Service, we have provided a set of example Dockerfiles for commonly used software stacks. These show examples of how to set up containers with non-root user access, as well as other best practices for developing secure containers. + +To request access to these containers please **ACTION TBD** + +## Example Containers + +| Software Stack | Container Name | Comments | +| --------------- | ---------------- | -------- | +| Freesurfer | freesurfer | | +| Jamovi | jamovi | | +| Julia | julia | | +| Jupyter Notebook | jupyter-docker-stack-basic | non-interactive at present | +| MinIO S3 | minioS3 | | +| Nextflow | nextflow | | +| NVIDIDA-Rapids | nvidia-rapids | basic/minimal packages | +| Octave | octave | | +| PostGreSQL | postgresql | | +| PSPP | pspp | | +| Python | python | | +| Pytorch | pytorch | | +| Quarto | quarto-jupyter, quarto-r | separate containers for R and Jupyter | +| Stata | stata | + +Most of these containers are minimum working examples, they are not fully fledged applications or workflow examples, but provided a template for setting up the technical parts of the containerisation process, such as user mapping, and mapping to any required `safe_data` folders or similar. diff --git a/mkdocs.yml b/mkdocs.yml index de2d354f2..e49e2b3fe 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -102,6 +102,7 @@ nav: - "Advising IG of Required Software Stack": safe-haven-services/tre-container-user-guide/advise-ig-required-software-stack.md - "Building and Testing Containers": safe-haven-services/tre-container-user-guide/building-and-testing-containers.md - "Using Containers in the TRE": safe-haven-services/tre-container-user-guide/using-containers-in-the-tre.md + - "Container Examples": safe-haven-services/tre-container-user-guide/container-examples.md - "EIDF Known Issues": known-issues/index.md - "EIDF Frequently Asked Questions": faq/index.md - "EIDF Tutorial Videos": tutorial-videos/index.md From cfa5e2152ecf7c7fc02cd0af72fdf6d39bcb5b5a Mon Sep 17 00:00:00 2001 From: Declan Valters Date: Wed, 18 Sep 2024 09:55:44 +0100 Subject: [PATCH 2/6] Add contact details --- .../container-examples.md | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/safe-haven-services/tre-container-user-guide/container-examples.md b/docs/safe-haven-services/tre-container-user-guide/container-examples.md index c63a27826..0049cf074 100644 --- a/docs/safe-haven-services/tre-container-user-guide/container-examples.md +++ b/docs/safe-haven-services/tre-container-user-guide/container-examples.md @@ -2,25 +2,25 @@ To help with writing your own Dockerfiles to run within a Trusted Research Environment via the Container Execution Service, we have provided a set of example Dockerfiles for commonly used software stacks. These show examples of how to set up containers with non-root user access, as well as other best practices for developing secure containers. -To request access to these containers please **ACTION TBD** +To request access to these container examples please contact Giulia Deiana or Declan Valters via the EIDF helpdesk. ## Example Containers -| Software Stack | Container Name | Comments | -| --------------- | ---------------- | -------- | -| Freesurfer | freesurfer | | -| Jamovi | jamovi | | -| Julia | julia | | -| Jupyter Notebook | jupyter-docker-stack-basic | non-interactive at present | -| MinIO S3 | minioS3 | | -| Nextflow | nextflow | | -| NVIDIDA-Rapids | nvidia-rapids | basic/minimal packages | -| Octave | octave | | -| PostGreSQL | postgresql | | -| PSPP | pspp | | -| Python | python | | -| Pytorch | pytorch | | -| Quarto | quarto-jupyter, quarto-r | separate containers for R and Jupyter | -| Stata | stata | +| Software Stack | Comments | +| --------------- | -------- | +| Freesurfer | | +| Jamovi | | +| Julia | | +| Jupyter Notebook | non-interactive at present | +| MinIO S3 | | +| Nextflow | | +| NVIDIDA-Rapids | basic/minimal packages | +| Octave | | +| PostGreSQL | | +| PSPP | | +| Python | | +| Pytorch | | +| Quarto | separate containers for R and Jupyter | +| Stata | Most of these containers are minimum working examples, they are not fully fledged applications or workflow examples, but provided a template for setting up the technical parts of the containerisation process, such as user mapping, and mapping to any required `safe_data` folders or similar. From 652762b932eab770e833febe94c426a4a0041928 Mon Sep 17 00:00:00 2001 From: Declan Valters Date: Thu, 26 Sep 2024 12:23:27 +0100 Subject: [PATCH 3/6] Pre-commit hook fixes --- .gitignore | 2 +- .../tre-container-user-guide/container-examples.md | 12 ++++++------ docs/tutorial-videos/index.md | 1 - 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/.gitignore b/.gitignore index 138da494a..fc25302ec 100644 --- a/.gitignore +++ b/.gitignore @@ -4,4 +4,4 @@ _build/ site/ .idea/ .envloc -mkdocs-material/ \ No newline at end of file +mkdocs-material/ diff --git a/docs/safe-haven-services/tre-container-user-guide/container-examples.md b/docs/safe-haven-services/tre-container-user-guide/container-examples.md index 0049cf074..dda4c21b5 100644 --- a/docs/safe-haven-services/tre-container-user-guide/container-examples.md +++ b/docs/safe-haven-services/tre-container-user-guide/container-examples.md @@ -1,12 +1,12 @@ # Container Examples -To help with writing your own Dockerfiles to run within a Trusted Research Environment via the Container Execution Service, we have provided a set of example Dockerfiles for commonly used software stacks. These show examples of how to set up containers with non-root user access, as well as other best practices for developing secure containers. +To help with writing your own Dockerfiles to run within a Trusted Research Environment via the Container Execution Service, we have provided a set of example Dockerfiles for commonly used software stacks. These show examples of how to set up containers with non-root user access, as well as other best practices for developing secure containers. To request access to these container examples please contact Giulia Deiana or Declan Valters via the EIDF helpdesk. ## Example Containers -| Software Stack | Comments | +| Software Stack | Comments | | --------------- | -------- | | Freesurfer | | | Jamovi | | @@ -16,11 +16,11 @@ To request access to these container examples please contact Giulia Deiana or De | Nextflow | | | NVIDIDA-Rapids | basic/minimal packages | | Octave | | -| PostGreSQL | | +| PostGreSQL | | | PSPP | | -| Python | | -| Pytorch | | +| Python | | +| Pytorch | | | Quarto | separate containers for R and Jupyter | | Stata | -Most of these containers are minimum working examples, they are not fully fledged applications or workflow examples, but provided a template for setting up the technical parts of the containerisation process, such as user mapping, and mapping to any required `safe_data` folders or similar. +Most of these containers are minimum working examples, they are not fully fledged applications or workflow examples, but provided a template for setting up the technical parts of the containerisation process, such as user mapping, and mapping to any required `safe_data` folders or similar. diff --git a/docs/tutorial-videos/index.md b/docs/tutorial-videos/index.md index dcc403712..31b687505 100644 --- a/docs/tutorial-videos/index.md +++ b/docs/tutorial-videos/index.md @@ -4,4 +4,3 @@ |--------| ------- | | Using the EIDF VM Service | Accessing the EIDF portal
Introduction to the project dashboard
Interfacing with the VM via the VDI (Remote Desktop)
Interfacing with the VM via the VDI (SSH)| | SSH-ing into an EIDF VM (Windows)
SSH-ing into an EIDF VM (Mac/Linux)| Generating an SSH key (Mac/Linux and Windows)
Registering an SSH public key with an EIDF project account
Setting up Multi-Factor Authentication
Verifying SSH access (Mac/Linux and Windows) | - From d27ec73623aedb254c80dfc23f3331c5d0fa21ca Mon Sep 17 00:00:00 2001 From: Declan Valters Date: Thu, 26 Sep 2024 12:25:45 +0100 Subject: [PATCH 4/6] Whitespace fixes --- .../building-and-testing-containers.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md b/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md index 28266b6c2..2c026bde5 100644 --- a/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md +++ b/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md @@ -2,7 +2,10 @@ ## Choose a container base from DockerHub -Projects should build containers by starting with a well-known application base container on a public registry. Projects should add a minimum of additional project software and packages so that the container is clearly built for a specific purpose. Containers built for one specific batch job, either a data transformation or analysis, are examples of this approach. Container builds that assemble groups of tools and then used to run a variety of tasks should be avoided. Additionally, container builds that start from generic distributions such as Debian or Ubuntu should also be avoided as leaner and more focussed application and language containers are already available. +Projects should build containers by starting with a well-known application base container on a public registry. Projects should add a minimum of additional project software and packages so that the container is clearly built for a specific purpose. +Containers built for one specific batch job, either a data transformation or analysis, are examples of this approach. +Container builds that assemble groups of tools and then used to run a variety of tasks should be avoided. +Additionally, container builds that start from generic distributions such as Debian or Ubuntu should also be avoided as leaner and more focussed application and language containers are already available. Examples of batch job container bases are Python and PyTorch, and other language specific and ML software stacks. Examples of interactive container bases are Rocker, Jupyter Docker Stacks, and NVIDIA RAPIDS extended with additional package sets and code required by your project. From 1161665eabca36a940aed4d3dea952f33e500b88 Mon Sep 17 00:00:00 2001 From: Declan Valters Date: Thu, 26 Sep 2024 12:47:02 +0100 Subject: [PATCH 5/6] Clarify Dockerfile Trivy scan vs image scan --- .../building-and-testing-containers.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md b/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md index 2c026bde5..c06ee9c62 100644 --- a/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md +++ b/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md @@ -53,7 +53,17 @@ If a model is downloaded from Hugging Face the advice is to set the environment It is recommended that the checklist for Dockerfile composition is followed: [Container Build Guide](https://github.com/EPCCed/tre-container-samples/blob/main/docs/container-build-guide.md) -Information Governance requirements may require a security scan of your container, and [Trivy](https://trivy.dev/) is a tool that can help with this task. Trivy inspects container images to find items which have known vulnerabilities and produces a report that may be used to help assess the risk. The use of the Trivy misconfiguration tool on Dockerfiles is also recommended. This tool option will highlight many common security issues: +Information Governance requirements may require a security scan of both: + +1. The Dockerfile used to build the container image + +1. The container image itself, once it is built. + +[Trivy](https://trivy.dev/) is a tool that can help with this task. Trivy inspects container images to find items which have known vulnerabilities and produces a report that may be used to help assess the risk. + +### 1. Scanning the container Dockerfile + +The use of the Trivy misconfiguration tool on Dockerfiles is also recommended. This tool option will highlight many common security issues in the Dockerfile: ```bash docker run --rm -v $(pwd):/repo ghcr.io/aquasecurity/trivy:latest config "/repo/Dockerfile" @@ -61,8 +71,8 @@ docker run --rm -v $(pwd):/repo ghcr.io/aquasecurity/trivy:latest config "/repo/ The security posture of containers and the build process may be of interest to IG teams, however, it is not expected that security issues indicated by the tool need to be addressed before the container is run in the TRE unless the IG team issues specific guidance on vulnerability and configuration remediation and mitigation. -## Scan container using Trivy CI +### 2. Scanning the container image using Trivy CI -Trivy can be run manually but it is easier to have it run automatically whenever you update your container image. An example GitHub Actions workflow to run Trivy and publish the outputs can be found [here](https://github.com/EPCCed/tre-container-samples/blob/main/.github/workflows/main.yaml) +Trivy can be run manually on the built image but it is easier to have it run automatically whenever you update your container image. An example GitHub Actions workflow to run Trivy and publish the outputs can be found [here](https://github.com/EPCCed/tre-container-samples/blob/main/.github/workflows/main.yaml) The Trivy report can be downloaded as an artifact from the job summary page. Before using a specific container in the TRE it may be necessary to test the security risk and gain IG team approval. From 1d8ea216b616ff2e089acf8f4eaab4ae83b8bc6c Mon Sep 17 00:00:00 2001 From: Declan Valters Date: Tue, 1 Oct 2024 14:45:24 +0100 Subject: [PATCH 6/6] Fix indentation bulleted list --- .../building-and-testing-containers.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md b/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md index c06ee9c62..3749829cf 100644 --- a/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md +++ b/docs/safe-haven-services/tre-container-user-guide/building-and-testing-containers.md @@ -55,11 +55,11 @@ It is recommended that the checklist for Dockerfile composition is followed: [Co Information Governance requirements may require a security scan of both: -1. The Dockerfile used to build the container image + 1. The Dockerfile used to build the container image -1. The container image itself, once it is built. + 1. The container image itself, once it is built. -[Trivy](https://trivy.dev/) is a tool that can help with this task. Trivy inspects container images to find items which have known vulnerabilities and produces a report that may be used to help assess the risk. +[Trivy](https://trivy.dev/) is a tool that can help with this task. Trivy inspects container images to find items which have known vulnerabilities and produces a report that may be used to help assess the risk. ### 1. Scanning the container Dockerfile