Is it possible to use wildcards in a bound singularity directory?

[davidaray]

I have created a container and want to analyze multiple files from the host system. I've bound the directory using --bind and can access single files but wildcards are not working to access all of them.

Is it possible to use wildcards when working with a mounted host directory?

If so, how?

I have tried:

$ singularity exec --bind data:/mnt gge_container_v4.sif fastqc -t 10 /mnt/read_files/SRR8482575_1.fastq.gz
application/gzip
Started analysis of SRR8482575_1.fastq.gz
Approx 5% complete for SRR8482575_1.fastq.gz
Approx 10% complete for SRR8482575_1.fastq.gz
Approx 15% complete for SRR8482575_1.fastq.gz
... and so on...

It works as expected.

However, there are three files in the directory and I want to analyze them all. Normally, when not using a container, the following will work. For example:

$ fastqc -t 10 data/read_files/*.fastq.gz
application/gzip
application/gzip
Started analysis of SRR8482575_1.fastq.gz
application/gzip
Started analysis of SRR8482575_2.fastq.gz
Started analysis of SRR8494924.fastq.gz
Approx 5% complete for SRR8482575_1.fastq.gz
Approx 5% complete for SRR8482575_2.fastq.gz
Approx 10% complete for SRR8482575_1.fastq.gz
Approx 10% complete for SRR8482575_2.fastq.gz
Approx 15% complete for SRR8482575_1.fastq.gz
Approx 5% complete for SRR8494924.fastq.gz
Approx 15% complete for SRR8482575_2.fastq.gz
... and so on...

But in this case, it does not.

$ singularity exec --bind data:/mnt gge_container_v4.sif fastqc -t 10 /mnt/read_files/*
Skipping '/mnt/read_files/*' which didn't exist, or couldn't be read

The eventual goal is to use this on many more files than just three. Calling each individually is not a viable option.

[dtrudg]

Wildcard patterns are evaluated by a shell. When you run a command on your host system fastqc -t 10 *.fastq.gz then your shell is evaluating the wild card, before it runs fastqc. The shell passes fastqc all of the matching file names when it runs it.

When you run the command in a container, using singularity exec, then there is no shell to evaluate the wildcard. It is passed directly to fastqc as-is.

You need to ensure:

• The wild card is quoted, so it is not evaluated by your host shell, before the singularity command is even run.
• Inside the container, your command is run with a shell, so it can evaluate the wildcards.

Try something like...

$ singularity exec --bind data:/mnt gge_container_v4.sif /bin/sh -c 'fastqc -t 10 /mnt/read_files/*'

[davidaray]

This works. However, it adds an increased level of complexity. I've created this container for a class I'm teaching and trying to keep things as simple as possible for the students, who will be a mixture of undergraduates with no coding experience and graduate students with varying levels of coding experience.

Adding the /bin/sh -c and quoting the command will introduce a new level of explanation that's needed. It's already a struggle with students to get them to simply use cp and understand directory structures when they've grown up using GUIs for everything.

Do you think it would be easier to bind the directory and then use the container as singularity shell gge_container_v4.sif ?

[dtrudg]

Yes, you could use singularity shell and then run the command inside the container shell. It might be easier to understand, but is a more interactive, manual method.

Other than using singularity shell, it will be necessary to at least quote the wildcard to make sure it isn't evaluated by the host shell. There isn't really a way around that, even if you create a container with a %runscript so that with singularity run it will evaluate the wildcard inside the container.

[davidaray]

Indeed, that does seem to solve the problem.

$ singularity shell --bind data:/mnt gge_container_v4.sif

Singularity> fastqc -t 10 /mnt/read_files/*
application/gzip
application/gzip
Started analysis of SRR8482575_1.fastq.gz
application/gzip
Started analysis of SRR8482575_2.fastq.gz
Started analysis of SRR8494924.fastq.gz
Approx 5% complete for SRR8482575_1.fastq.gz
Approx 5% complete for SRR8482575_2.fastq.gz
Approx 10% complete for SRR8482575_1.fastq.gz
... and so on...

I'll experiment with this and see if I can get it to work for the students.

Thank you for the help. This inspired a potential solution.