Running metaGEM.sh parser locally

[karinedurand]

Hi,
I install metagem automatically on my local computer. I have no problem with bash metaGEM.sh -t downloadToy and bash metaGEM.sh -t organizeData in dataset folder. But I have an error in the nohup.out :
+1: Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 2
Job counts:
count jobs
1 all
1
Select jobs to execute...

[Wed Mar 3 12:56:47 2021]
Job 0:
WARNING: Be very careful when adding/removing any lines above this message.
The metaGEM.sh parser is presently hardcoded to edit line 22 of this Snakefile to expand target rules accordingly,
therefore adding/removing any lines before this message will likely result in parser malfunction.

/bin/sh: 1: sbatch: not found
Error submitting jobscript (exit code 127):

Job failed, going on with independent jobs.

Do you know what can i do ?
Best

[franciscozorrilla]

Hi Karine,

You are getting that error because your machine does not recognize the Slurm cluster command sbatch. The metaGEM.sh parser is mainly designed for interfacing with a high performance computer cluster, as its main task is to submit jobs to the scheduler. The organizeData and downloadToy rules worked for you because they are submitted locally (i.e. not as a job but would run on the login node of the cluster). The pipeline interface was designed with cluster usage in mind considering the large sizes of metagenomic datasets and the computational resources required for assembling short reads from such datasets. In theory you should be able to run metaGEM on your local machine by appropriately modifying the rule all input on line 22, but this will likely not be very practical outside of running small subsets/toy examples. Do you have access to your institution's high performance computer cluster? I would definitely recommend running metaGEM on the cluster for a real metagenomic dataset.

For the sake of explanation, if you wanted to run quality filtering jobs on your local computer then you would replace line 22 of your Snakefile with the output of the qfilter rule in the Snakefile, i.e. :
expand(f'{config["path"]["root"]}/{config["folder"]["qfiltered"]}/{{IDs}}/{{IDs}}_R1.fastq.gz', IDs=IDs) (line 150). Then you can try running snakemake all -n to have a test dry-run to see if the wildcards are being expanded correctly based on your sample IDs. This editing/modification of the rule all input on line 22 is part of the job of the metaGEM.sh parser, but I essentially hardcoded which tasks should be submitted locally and which tasks should be submitted to the cluster. If it is helpful for you I can add an option to the metaGEM.sh parser to force it to submit certain jobs/tasks locally.

Thanks for your interest in metaGEM and please let me know if you have any follow up questions.
Best wishes,
Francisco

[kunaljaani]

Hi Francisco,

Thank you for your efforts and for sharing the metaGEM.
my query is regarding the same topic. I used the solution provided here to run metaGEM locally (with 16 cores and 128Gb RAM); snakemake all -n works well but I still get the same error. Could you please suggest me some way out?

Thank you very much.

[franciscozorrilla]

Hi Kunal,

Thanks for trying out metaGEM! Based on the second image you posted, it appears that the error is caused because conda could not find the envs/metagem environment. One way to fix this issue would be to just hardcode your metagem env path into the Snakefile rules. Please let me know if this solves your issue or if you run into any further problems!

Best,
Francisco

[kunaljaani]

Hi Francisco,

Thank you for your rapid response. I'll make the suggested changes and I hope that will work for me. In any case I'll update you.

Thanks a lot.
Kunal

[karinedurand]

I thought it could work on a computer because in the automatic installation it was indicated that it was possible to install it on a local machine.
So thanks for the Flag!
Best
Karine

[Lucas-Maciel]

Hi @franciscozorrilla,

I work with an HPC that doesn't use Slurm to run our jobs, so this flag would be useful for HPCs as well.

[franciscozorrilla]

Hi @karinedurand, yes you can install it locally for running small tests or a small subset of samples, but I want to highlight the fact that it will not be practical for your to process entire real metagenomic datasets (on the order of terabytes of data) on your local machine. I would strongly recommend you obtain access to a computer cluster for analysis of metagenomic data with metaGEM.

Hi @Lucas-Maciel, what workload manager does your cluster use? We can probably easily modify the metaGEM.sh parser to have a flag (e.g. --LSF) for submitting jobs in clusters that have your workload manager. This is probably a better solution for you since you wouldn't want to be running all your jobs on the login node of the cluster.

Best,
Francisco

[Lucas-Maciel]

@franciscozorrilla our HPC is using SGE.

[franciscozorrilla]

@Lucas-Maciel, OK thanks for the info. I confess that have never used SGE/OGE, and I do not have access to a cluster with that workload manager, so I may need your help to double check that everything is running smoothly. Based on some documentation that I am reading, it seems like it should be quite trivial to add support for SGE/OGE. If I understand correctly qsub is the SGE/OGE equivalent of sbatch? You could try replacing every occurrence of sbatch with qsub in the metaGEM.sh file (e.g. sed -i 's/sbatch/qsub/g' metaGEM.sh on the command line, or using the find and replace function of your favorite text editor). If you could try that for me and let me know that everything works as expected then I can modify the metaGEM.sh parser to read in a workload scheduler flag. I will also add the option to submit locally/on the login node.

[Lucas-Maciel]

@franciscozorrilla I replaced it with sed but it will still need some modifications regarding the configuration files (how to set memory, time, how many jobs at a time...). I'll try to do that myself

qsub: Negative or zero step in range is not allowed
Error submitting jobscript (exit code 7):

[franciscozorrilla]

Hi @karinedurand, I have modified the metaGEM.sh script in this commit to take the -l or --local flag for running jobs on a local machine. Simply replace your metaGEM.sh script with the newer version on the master branch and give it a try. You don't have to specify number of jobs, number of cores, memory, or hours, since these parameters are only used by the cluster workload scheduler for submitting jobs. Your jobs will run for as long as they need to (likely for a long time in the case of assemblies for example), they will run in series (i.e. only one at a time), and they will use as much memory as is available on your machine. Regarding the number of cores for the different tasks, this should still be specified in your config.yaml file. For example, you can try running bash metaGEM.sh -t fastp -l. You can check the file nohup.out to see if you tasks are running successfully. I ran some small tests and it seems to be working as expected, but let me know if you run into problems.

@Lucas-Maciel let me know if/when you are able to modify the cluster_config.json/metaGEM.sh files to play nice the the SGE scheduler and I will gladly incorporate your changes. I opened up a new issue (#18) specific to this.

Best wishes,
Francisco

[franciscozorrilla]

Closing this for now due to inactivity.
Please re-open if any issues arise.