Incorporating and streamlining checkpoint update

README.md

@@ -1,24 +1,20 @@
-# SAMSA2 - A fork of the complete metatranscriptome analysis pipeline
+# SAMSA2 - A complete metatranscriptome analysis pipeline
 
-### modification by sebastien.renaut@gmail.com
+*****
 
-### New in my version. 
-This is a new master script (master_samsa2.slurm) with several modifications:
+**Version 2.2.0 - Yesod - Modifications added by sebastien.renaut@gmail.com:**
 
-* It specifies multithreading for all programs for which it is available (trimmomatic, pear, diamond)
-* The script automatically creates a `checkpoint` file. Once a step is finished, it writes the name of that specific step in `checkpoint`. Everytime you run the script, it looks if that file exist. If so, it reads it and SKIP the steps written in checkpoint. This is done to avoid re-run CPU intense steps if not necessary.
-* In the merging step, unmerged reads are concatenated and added to a single file. The forward read and the reverse (complement) read are concatenated with a string of 20 Ns in the middle: This is done through a new R script entitled: `combining_umerged.R`
+* Multithreading added for PEAR, Trimmomatic, SortMeRNA
+* The script automatically creates a `checkpoint` file. Once a step is finished, it writes the name of that specific step in `checkpoint` and that step is skipped on a rerun of the master_script. This is done to avoid re-running CPU-intensive steps if unnecessary.
+* A new version of the master script now exists, called "master\_script\_preserving\_unmerged.sh".  In this script, in the merging step, unmerged reads are concatenated and added to a single file. The forward read and the reverse (complement) read are concatenated with a string of 20 Ns in the middle: This is done through a new R script entitled: `combining_umerged.R`
 * Extra care is taken to remove unnecessary files once a step is performed to keep disk usage at a minimum.
 * Each step contains an exit statement to be printed if the master script dies due to an unforseen error.
 * Trimmomatic removes adapter contamination according to a specific fasta file.
 * All options, read & program location are to be specified in the first section of the script.
 * The script is formated to be run on a HPC using a SLURM job scheduler, but this can be easily changed / removed.
-* I've removed the --num_alignments 0 in the ribosomal `sortmrna` step. This caused problems and slowed things down a lot. Plus we don't care about the rRNA alignments. If a sequence aligns to 1 or 1,000 rRNA its out anyways...
-
-#### To do:
-* reverse the order of the merging and trimming step (but again, maybe not...)
-* simplify some syntax / make sure all paths are relative (mostly done)
+* The flag --num_alignments 0 in the ribosomal `sortmrna` step has been removed. This caused problems and slowed things down a lot. Plus, we don't care about the rRNA alignments - whether a sequence aligns to 1 or 1,000 rRNA, it's out anyways...
 
+*****
 
 Version 2 of the SAMSA pipeline - faster!  Lighter!  More options!  Less waiting!  
 
@@ -32,7 +28,7 @@ Version 2 of the SAMSA pipeline - faster!  Lighter!  More options!  Less waiting
 
 ### Dependencies
 
-SAMSA2 requires Python2 for aggregation scripts.  Currently, this pipeline does not work with Python3, although an update is planned.
+SAMSA2 requires Python2 for aggregation scripts.  Currently, this pipeline works mostly with Python3, although there may be some errors not yet caught.
 
 The following programs can be downloaded OR can be installed from the binaries provided in the programs/ folder.
 

R_scripts/combining_umerged.R

@@ -1,24 +1,23 @@
 #!/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/r/3.5.0/bin/Rscript
 
 args = commandArgs(TRUE)
-start = args[1] # Specify which sequences in "list_ind" file you want to align, directlty from the shell. Alternatively, you can do this from the "alignments" function itself.
-#start="/Users/jerry/Documents/CSBQ/shapiro/results"
+start = args[1] # Specify which sequences in "list_ind" file you want to align, directly from the shell. Alternatively, you can do this from the "alignments" function itself.
 
 setwd(start)
 
-###files to work with 
+###files to work with
 system(paste("ls -1 *unassembled.forward.fastq >umerged.forward.files",sep = ""))
 system(paste("ls -1 *unassembled.reverse.fastq >umerged.reverse.files",sep = ""))
 system(paste("ls -1 *.assembled.fastq >assembled.files",sep = ""))
 
-files_f = read.table("umerged.forward.files",stringsAsFactors = F)      
-files_r = read.table("umerged.reverse.files",stringsAsFactors = F) 
-files_a = read.table("assembled.files",stringsAsFactors = F) 
+files_f = read.table("umerged.forward.files",stringsAsFactors = F)
+files_r = read.table("umerged.reverse.files",stringsAsFactors = F)
+files_a = read.table("assembled.files",stringsAsFactors = F)
 
 
-for(i in 1:nrow(files_f))      
+for(i in 1:nrow(files_f))
   {
-#grep 1st and 2rd lines 
+#grep 1st and 2rd lines
   	unassembled.fastq = paste("awk 'NR % 2 == 1' ",files_f[i,1]," >unassembled.names.seq.fastq",sep="")
   	system(unassembled.fastq)
 
@@ -33,9 +32,9 @@ for(i in 1:nrow(files_f))
 #N and E quality file
  	system("wc -l unassembled.seq.reverse.fastq >wc")
  	wc = read.table("wc")
-  
+
  	write.table(c(rbind(rep('NNNNNNNNNNNNNNNNNNNN',wc[1,1]/2),rep('EEEEEEEEEEEEEEEEEEEE',wc[1,1]/2))),"NE_file",row.names = F, col.names = F, quote = F)
-  
+
 #paste the unassembled sequences into a single file
 	system("paste -d '\\0' unassembled.seq.forward.fastq NE_file unassembled.seq.reverse.fastq >unassembledN")
 
@@ -54,8 +53,3 @@ for(i in 1:nrow(files_f))
 
 #remove the clutter (listing of all files)
 system("rm assembled.files umerged.forward.files umerged.reverse.files")
-
-
-
-
-