Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EVA-3663_cli_Documentation #59

Merged
merged 7 commits into from
Sep 17, 2024
Merged

EVA-3663_cli_Documentation #59

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9d31d03
test commit
Dona094 Sep 16, 2024
4eea33e
test commit
Dona094 Sep 17, 2024
b8b1ae7
Update Getting_Started_with_eva_sub_cli.md
Dona094 Sep 17, 2024
28fcb83
Update Getting_Started_with_eva_sub_cli.md
Dona094 Sep 17, 2024
c1a17e1
Apply suggestions from code review
Dona094 Sep 17, 2024
431e3af
Update README.md
Dona094 Sep 17, 2024
5ca169a
Apply suggestions from code review
Dona094 Sep 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Getting_Started_with_eva_sub_cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# How to generate inputs for the eva-sub-cli
# Comprehensive overview of the required inputs for eva-sub-cli tool

The eva-sub-cli tool requires the following inputs:

Expand Down Expand Up @@ -56,7 +56,7 @@ The CLI tool performs the following validation checks and generates correspondin

In the following sections, we will examine each of these checks in detail, starting with the Metadata check.

## Metadata check
# Metadata check

Once the user passes the metadata spreadsheet for validation checks, the eva-sub-cli tool verifies that all mandatory columns, marked in bold in the spreadsheet, are filled in. This data is crucial for further validation processes, such as retrieving the INDSC accession of the reference genome used to call the variants, and for sample and project metadata. If any mandatory columns or sheets are missing, the CLI tool will raise errors.

Expand Down
88 changes: 53 additions & 35 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,31 +1,35 @@
# eva-sub-cli
EVA Submission Command Line Interface for Validation
The eva-sub-cli tool is a command line interface tool for data validation and upload. The tool transforms the submission process at EVA by enabling users to take control of data validation process. Previously handled by our helpdesk team, validation can now be performed directly by users, streamlining and improving the overall submission workflow at the EVA.


## Installation

There are currently three ways to install and run the tool: using conda, from source using Docker,
and from source natively (i.e. managing dependencies on your own).
There are currently three ways to install and run the tool :
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently there's some cultural differences in spacing for colons... I don't mind either way, as long as we're consistent.

- Using conda
- From source using Docker
- From source natively (i.e. managing dependencies on your own)
Dona094 marked this conversation as resolved.
Show resolved Hide resolved

### Conda
### 1. Conda

The most straightforward way to install eva-sub-cli and its dependencies is through conda.
For example the following installs eva-sub-cli in a new environment called `eva`, activates the environment, and prints
For instance, the following commands install eva-sub-cli in a new environment called `eva`, activate the environment, and print
the help message:
```bash
conda create -n eva -c conda-forge -c bioconda eva-sub-cli
conda activate eva
eva-sub-cli.py --help
````

### From source using Docker
### 2. From source using Docker

Docker provides an easy way to run eva-sub-cli without installing dependencies separately.
This method requires just Python 3.8+ and [Docker](https://docs.docker.com/engine/install/) to be installed.
Then either clone the git repository, or download the newest tagged release from [here](https://github.com/EBIvariation/eva-sub-cli/tags):
Once it is set up, you can either clone the git repository, or download the newest tagged release from [here](https://github.com/EBIvariation/eva-sub-cli/tags):
```bash
git clone git@github.com:EBIvariation/eva-sub-cli.git
git clone https://github.com/EBIvariation/eva-sub-cli.git

# OR (replace "v0.2" with the newest version)

wget -O eva-sub-cli.zip https://github.com/EBIvariation/eva-sub-cli/archive/refs/tags/v0.2.zip
unzip eva-sub-cli.zip && mv eva-sub-cli-* eva-sub-cli
```
Expand All @@ -37,36 +41,44 @@ cd eva-sub-cli
python -m pip install .
```

To check it is installed correctly, you can run:
To verify that the cli tool is installed correctly, run the following command, and you should see the help message displayed :
```bash
eva-sub-cli.py -h
```

### From source natively
### 3. From source natively

This method requires the following:
This installation method requires the following :
* Python 3.8+
* [Nextflow](https://www.nextflow.io/docs/latest/getstarted.html) 21.10+
* [biovalidator](https://github.com/elixir-europe/biovalidator) 2.1.0+
* [vcf-validator](https://github.com/EBIvariation/vcf-validator) 0.9.6+
Dona094 marked this conversation as resolved.
Show resolved Hide resolved

Install each of these and ensure they are available on the path.
Then git clone the repo or install the newest release as described above.
Install each of these and ensure they are included in your PATH. Then, either clone the repository using Git or install the latest release as previously described.

## Getting started with the eva-sub-cli tool

## Input files for the validation and submission tool

There are two ways of specifying the VCF files and associated reference genome
The ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.

### Using `--vcf_files` and `--reference_fasta`
## eva-sub-cli tool: Options and parameters guide

This allows you to provide multiple VCF files to validate and a single associated reference genome file.
The VCF files and the associated reference genome file must use the same chromosome naming convention
The eva-sub-cli tool provides several options/parameters that you can use to tailor its functionality to your needs. Understanding these parameters is crucial for configuring the tool correctly. Below is an overview of the key parameters and options:

### Using metadata file by providing `--metadata_json` or `--metadata_xlsx`
| OPTIONS/PARAMETERS | DESCRIPTION |
|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| --version | Shows version number of the program and exit |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically I've edited the --help text in the code... we can transfer those edits here but it's not a big deal.

| --metadata_xlsx | Excel spreadsheet that describe the project, analysis, samples and files |
| --metadata_json | Json file that describe the project, analysis, samples and files |
| --vcf_files | One or several vcf files to validate.This allows you to provide multiple VCF files to validate and a single associated reference genome file. The VCF files and the associated reference genome file must use the same chromosome naming convention |
| --reference_fasta | The fasta file containing the reference genome from which the variants were derived |
| --submission_dir | Path to the directory where all processing will be done and submission data is/will be stored |
| --tasks {validate,submit} | Selecting VALIDATE will run the validation regardless of the outcome of previous runs. Selecting SUBMIT will run validate only if the validation was not performed successfully before and then run the submission |
| --executor {docker,native} | Select an execution type for running validation (default native) |
| --shallow | Set the validation to be performed on the first 10000 records of the VCF. Only applies if the number of record exceed 10000 |
| --username | Username used for connecting to the ENA webin account |
| --password | Password used for connecting to the ENA webin account |

The path to the VCF files are provided in the Files section of the metadata and their corresponding fasta sequence is provided in the analysis section.
This allows us to support different assemblies for each VCF file.
Please check the below sections `The metadata spreadsheet` and `The metadata JSON` for the format and options available in metadata files.

### The metadata spreadsheet

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edit for the following lines:
The metadata template can be found within the [etc folder](eva_sub_cli/etc/EVA_Submission_template.xlsx). It should be populated following the instruction provided within the template.

Expand All @@ -75,16 +87,28 @@ It should be populated following the instruction provided within the template

### The metadata JSON

The metadata can also be provided via a JSON file which should conform to the schema located at
`eva_sub_cli/etc/eva_schema.json`
The metadata can also be provided via a JSON file, which should conform to the schema located [here](eva_sub_cli/etc/eva_schema.json).

More detail documentation to follow

## Execution

**Note for Docker users:** for each of the below commands, add the command line option `--executor docker`, which will
fetch and manage the docker container for you. Make sure that Docker is running in the background, e.g.
by opening Docker Desktop.
### Validate only

To validate and not submit, run the following command:

```shell
eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE
```

**Note for Docker users:**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the logic of putting this within the validate section - it's really needed for both, because submit will run validation if it hasn't been run before. But if you think that's still clear from this placement then I guess it's fine.


Make sure that Docker is running in the background, e.g. by opening Docker Desktop.
For each of the below commands, add the command line option `--executor docker`, which will
fetch and manage the docker container for you.

```shell
eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE --executor docker
```

### Validate and submit your dataset

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edit for the next line:
To validate and submit, run the following command:

Expand All @@ -95,16 +119,10 @@ eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx \
--vcf_files vcf_file1.vcf vcf_file2.vcf --reference_fasta assembly.fa --submission_dir submission_dir
```

### Validate only

To validate and not submit run the following command

```shell
eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE
```
### Submit only

All submission must have been validated. You cannot run the submission without validation. Once validated running
All submissions must have been validated. You cannot run the submission without validation. Once validated, execute the following command:

```shell
eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edit for the last line of the file:
This will only submit the data and not validate.

Expand Down
Loading