rsempipeline is a pipeline for analyzing GEO data using RSEM. The typical analysis process is as follows:
The input to the pipeline are mainly from two resources,
- soft files for all Series (aka. GSE)
- A GSE_species_GSM.csv file which contains a list of all interested samples (aka. GSM) to be processed
There are three steps included in this pipeline:
- Download the sra files for all GSMs from GEO website using aspc from Aspera or wget (in case when aspc fails). aspc and wget use different urls which are linked to copies of the same file.
- sra files are converted to fastq.gz files using fastq-dump from SRA Toolkit
- Run rsem-calculate-expression from RSEM package with all fastq.gz files for all GSMs
The pipeline is designed to run the first two steps (computationally cheap) on a localhost. Step 3 (computationally expensive) is run on a HPC cluster (e.g. genesis, westgrid cluster).
Typically, about 100 GSEs and a few thousands of GSMs are picked by our
collaborators and grouped into a batch. Step 1 and 2 are done in a
sub-batch-by-sub-batch fashion where all GSMs of a sub-batch are processed in
parallel until finished. Each sub-batch of GSMs are selected based on their
file sizes (estimated from sizes of sra and its resultant fastq.gz files) and
how much disk space available on the localhost as specified in a configuration
file (rp_config.yml). At the end of the second step, a submission script
will be generated for each GSM, and at Step 3 a new job will be submitted to
the cluster for processing the GSM using RSEM. A control mechanism has also
been implemented to avoid overuse of the cluster resources such as compute
nodes and disk space. The first two steps are run by the command rp-run
while the generation of the submission script and job submission are handled by
the command rp-transfer.
For installation and usage instructions, please refer to INSTALL.rst and
USAGE.rst.
If you have found any bugs, questions, comments, please contact Zhuyi Xue (zxue@bcgsc.ca).