singularity build --sandbox: Different run --writable behaviours w.r.t. to the use of sudo

[ReHoss]

Hi everyone,

I have the following questions regarding the use of sudo to build images:

On my local machine, I have two choices when building (w.r.t. being sudo):

singularity build --sandbox firedrake-vanilla docker://firedrakeproject/firedrake-vanilla
sudo singularity build --sandbox firedrake-vanilla docker://firedrakeproject/firedrake-vanilla

Then, I have two choices when running:

singularity run --writable firedrake-vanilla
sudo singularity run --writable firedrake-vanilla

The build is based on the Docker image. The image start with USER root but switch to a newly created USER firedrake later.

I observe different behaviours with respect to the four distinct build/run combinations:

sudo build/sudo run --writable : /home/firedrake exists, ls is authorised (sandbox image is owned by "root")
sudo build/run --writable: /home/firedrake does not exist, (sandbox image is owned by root)

build/sudo run --writable: /home/firedrake exists, ls is authorised (sandbox image is owned by $USER)
build/run --writable: /home/firedrake does not exist, (sandbox image is owned by $USER)

• Is it possible to get the logs of the building process to see if the image construction is going well, step by step?

• Is it possible to have a build process where a failing instruction is ignored, letting the building process continue until the end of the image instructions file? Indeed, I thought the absence of /home/firedrake was due to a failed build.

• What are the associated parts of the official documentation where the sudo behaviour is explained (except the security part)?

• What are the advantages of building without sudo ? For a standard one-user workflow, is it the best practice to build as sudo locally and run without sudo anywhere?

• Why do I see 2024/02/01 12:25:09 warn rootless [...] , when I build without sudo ?

Thank you a lot for your help,
Best,

[ReHoss]

Some answers from @dtrudg on Slack:

The main thing to understand, before going forward with some of the other questions, is that by default Singularity does not honour the USER in the Dockerfile. When you run the container you are not the firedrake user.
You may want to review this part of the guide, first… https://docs.sylabs.io/guides/4.0/user-guide/singularity_and_docker.html#differences-and-limitations-vs-docker
You may also want to look at the new OCI mode, which does honour USER… https://docs.sylabs.io/guides/4.0/user-guide/oci_runtime.html#oci-mode-oci

Right, okay… when you run without sudo, your own home directory from the host system is going to be mounted in the container. It’s probably under /home - and the way Singularity is binding stuff for --writable mode with a sandbox means that all of /home in the container image is going to be masked.
It doesn’t happen when you run with sudo because root’s home is at /root

Okay, yeah. in writable mode the bind mounting is working differently on a sandbox, because of the combination of filesystems making up the container.

Add --no-mount home and you should see it again…
$ singularity run --writable --no-mount home firedrake-vanilla
Singularity> ls /home
firedrake
(but you won’t get your own home directory then)

Here: https://docs.sylabs.io/guides/latest/user-guide/singularity_and_docker.html#default-mounts-home

Software inside another user’s home directory, e.g. /home/myapp, may be obscured by SingularityCE’s automatic mounts onto /home.
We avoid the specific detail of --writable and other circumstances… but it’s covered in best practice for Docker/Singularity compatibility there.

If you are working with sandboxes, general advice would be build as the same user you run them as.

• If you build as sudo, run with sudo
• If you build without sudo, run without sudo

A sandbox isn’t intended to be absolutely transferrable between users. Because it is just files on the host then it is affected by permissions / ACLs / any issues with network filesystems etc.

[ReHoss]

Alright, as pointed out by @dtrudg, the container folder is obscured because of the mounting process. It was not a building problem or anything.
source: https://docs.sylabs.io/guides/latest/user-guide/singularity_and_docker.html#default-mounts-home