Skip to content

Commit

Permalink
Merge pull request #59 from EBIvariation/EVA3663_Documentation
Browse files Browse the repository at this point in the history 
EVA-3663_cli_Documentation
  • Loading branch information
@apriltuesday
apriltuesday authored Sep 17, 2024
2 parents 5222976 + 5ca169a commit 835012b
Show file tree
Hide file tree
Showing 2 changed files with 56 additions and 38 deletions.
4 changes: 2 additions & 2 deletions Getting_Started_with_eva_sub_cli.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# How to generate inputs for the eva-sub-cli
# Comprehensive overview of the required inputs for eva-sub-cli tool

The eva-sub-cli tool requires the following inputs:

Expand Down Expand Up @@ -56,7 +56,7 @@ The CLI tool performs the following validation checks and generates correspondin

In the following sections, we will examine each of these checks in detail, starting with the Metadata check.

## Metadata check
# Metadata check

Once the user passes the metadata spreadsheet for validation checks, the eva-sub-cli tool verifies that all mandatory columns, marked in bold in the spreadsheet, are filled in. This data is crucial for further validation processes, such as retrieving the INDSC accession of the reference genome used to call the variants, and for sample and project metadata. If any mandatory columns or sheets are missing, the CLI tool will raise errors.

Expand Down
90 changes: 54 additions & 36 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,31 +1,35 @@
# eva-sub-cli
EVA Submission Command Line Interface for Validation
The eva-sub-cli tool is a command line interface tool for data validation and upload. The tool transforms the submission process at EVA by enabling users to take control of data validation process. Previously handled by our helpdesk team, validation can now be performed directly by users, streamlining and improving the overall submission workflow at the EVA.


## Installation

There are currently three ways to install and run the tool: using conda, from source using Docker,
and from source natively (i.e. managing dependencies on your own).
There are currently three ways to install and run the tool :
- Using conda
- From source using Docker
- From source natively (i.e. installing dependencies yourself)

### Conda
### 1. Conda

The most straightforward way to install eva-sub-cli and its dependencies is through conda.
For example the following installs eva-sub-cli in a new environment called `eva`, activates the environment, and prints
For instance, the following commands install eva-sub-cli in a new environment called `eva`, activate the environment, and print
the help message:
```bash
conda create -n eva -c conda-forge -c bioconda eva-sub-cli
conda activate eva
eva-sub-cli.py --help
````

### From source using Docker
### 2. From source using Docker

Docker provides an easy way to run eva-sub-cli without installing dependencies separately.
This method requires just Python 3.8+ and [Docker](https://docs.docker.com/engine/install/) to be installed.
Then either clone the git repository, or download the newest tagged release from [here](https://github.com/EBIvariation/eva-sub-cli/tags):
Once it is set up, you can either clone the git repository, or download the newest tagged release from [here](https://github.com/EBIvariation/eva-sub-cli/tags):
```bash
git clone git@github.com:EBIvariation/eva-sub-cli.git
git clone https://github.com/EBIvariation/eva-sub-cli.git
# OR (replace "v0.2" with the newest version)
wget -O eva-sub-cli.zip https://github.com/EBIvariation/eva-sub-cli/archive/refs/tags/v0.2.zip
unzip eva-sub-cli.zip && mv eva-sub-cli-* eva-sub-cli
```
Expand All @@ -37,36 +41,44 @@ cd eva-sub-cli
python -m pip install .
```

To check it is installed correctly, you can run:
To verify that the cli tool is installed correctly, run the following command, and you should see the help message displayed :
```bash
eva-sub-cli.py -h
```

### From source natively
### 3. From source natively

This method requires the following:
This installation method requires the following :
* Python 3.8+
* [Nextflow](https://www.nextflow.io/docs/latest/getstarted.html) 21.10+
* [biovalidator](https://github.com/elixir-europe/biovalidator) 2.1.0+
* [vcf-validator](https://github.com/EBIvariation/vcf-validator) 0.9.6+
* [vcf-validator](https://github.com/EBIvariation/vcf-validator) 0.9.7+

Install each of these and ensure they are included in your PATH. Then, either clone the repository using Git or install the latest release as previously described.

Install each of these and ensure they are available on the path.
Then git clone the repo or install the newest release as described above.
## Getting started with the eva-sub-cli tool

## Input files for the validation and submission tool

There are two ways of specifying the VCF files and associated reference genome
The ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.

### Using `--vcf_files` and `--reference_fasta`
## eva-sub-cli tool: Options and parameters guide

This allows you to provide multiple VCF files to validate and a single associated reference genome file.
The VCF files and the associated reference genome file must use the same chromosome naming convention
The eva-sub-cli tool provides several options/parameters that you can use to tailor its functionality to your needs. Understanding these parameters is crucial for configuring the tool correctly. Below is an overview of the key parameters and options:

### Using metadata file by providing `--metadata_json` or `--metadata_xlsx`
| OPTIONS/PARAMETERS | DESCRIPTION |
|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| --version | Shows version number of the program and exit |
| --metadata_xlsx | Excel spreadsheet that describe the project, analysis, samples and files |
| --metadata_json | Json file that describe the project, analysis, samples and files |
| --vcf_files | One or several vcf files to validate.This allows you to provide multiple VCF files to validate and a single associated reference genome file. The VCF files and the associated reference genome file must use the same chromosome naming convention |
| --reference_fasta | The fasta file containing the reference genome from which the variants were derived |
| --submission_dir | Path to the directory where all processing will be done and submission data is/will be stored |
| --tasks {validate,submit} | Selecting VALIDATE will run the validation regardless of the outcome of previous runs. Selecting SUBMIT will run validate only if the validation was not performed successfully before and then run the submission |
| --executor {docker,native} | Select an execution type for running validation (default native) |
| --shallow | Set the validation to be performed on the first 10000 records of the VCF. Only applies if the number of record exceed 10000 |
| --username | Username used for connecting to the ENA webin account |
| --password | Password used for connecting to the ENA webin account |

The path to the VCF files are provided in the Files section of the metadata and their corresponding fasta sequence is provided in the analysis section.
This allows us to support different assemblies for each VCF file.
Please check the below sections `The metadata spreadsheet` and `The metadata JSON` for the format and options available in metadata files.

#### The metadata spreadsheet

Expand All @@ -75,16 +87,28 @@ It should be populated following the instruction provided within the template

#### The metadata JSON

The metadata can also be provided via a JSON file which should conform to the schema located at
`eva_sub_cli/etc/eva_schema.json`
The metadata can also be provided via a JSON file, which should conform to the schema located [here](eva_sub_cli/etc/eva_schema.json).

More detail documentation to follow

## Execution

**Note for Docker users:** for each of the below commands, add the command line option `--executor docker`, which will
fetch and manage the docker container for you. Make sure that Docker is running in the background, e.g.
by opening Docker Desktop.
### Validate only

To validate and not submit, run the following command:

```shell
eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE
```

**Note for Docker users:**

Make sure that Docker is running in the background, e.g. by opening Docker Desktop.
For each of the below commands, add the command line option `--executor docker`, which will
fetch and manage the docker container for you.

```shell
eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE --executor docker
```

### Validate and submit your dataset

Expand All @@ -95,16 +119,10 @@ eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx \
--vcf_files vcf_file1.vcf vcf_file2.vcf --reference_fasta assembly.fa --submission_dir submission_dir
```

### Validate only

To validate and not submit run the following command

```shell
eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE
```
### Submit only

All submission must have been validated. You cannot run the submission without validation. Once validated running
All submissions must have been validated. You cannot run the submission without validation. Once validated, execute the following command:

```shell
eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir
Expand Down

0 comments on commit 835012b

Please sign in to comment.