From 571449d504b31ab601f9824728269638b5c4ef41 Mon Sep 17 00:00:00 2001 From: bedroesb Date: Thu, 7 Mar 2024 19:33:03 +0100 Subject: [PATCH 01/12] start with doc --- _data/sidebars/main.yml | 2 + _sass/_bootstrap_variables.scss | 1 + images/datahub-to-ena.svg | 330 ++++++++++++++++++++++++++++++ images/ena-submission-toolbox.svg | 229 +++++++++++++++++++++ pages/ena-submission.md | 92 +++++++++ 5 files changed, 654 insertions(+) create mode 100644 images/datahub-to-ena.svg create mode 100644 images/ena-submission-toolbox.svg create mode 100644 pages/ena-submission.md diff --git a/_data/sidebars/main.yml b/_data/sidebars/main.yml index 0f30fbb..3c3fe46 100644 --- a/_data/sidebars/main.yml +++ b/_data/sidebars/main.yml @@ -76,6 +76,8 @@ subitems: url: /uniprot - title: ArrayExpress url: /arrayexpress + - title: ENA data submission toolbox + url: /ena-submission - title: Covid-19 data submission url: /covid-19/ subitems: diff --git a/_sass/_bootstrap_variables.scss b/_sass/_bootstrap_variables.scss index 087bdf5..ca64c8d 100644 --- a/_sass/_bootstrap_variables.scss +++ b/_sass/_bootstrap_variables.scss @@ -30,4 +30,5 @@ $h3-font-size: 1rem * 1.5 !default; $h4-font-size: 1rem * 1.1 !default; $h5-font-size: 1rem !default; $h6-font-size: 1rem !default; +$table-bg: $white; diff --git a/images/datahub-to-ena.svg b/images/datahub-to-ena.svg new file mode 100644 index 0000000..f44ada7 --- /dev/null +++ b/images/datahub-to-ena.svg @@ -0,0 +1,330 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/images/ena-submission-toolbox.svg b/images/ena-submission-toolbox.svg new file mode 100644 index 0000000..87e9a56 --- /dev/null +++ b/images/ena-submission-toolbox.svg @@ -0,0 +1,229 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/pages/ena-submission.md b/pages/ena-submission.md new file mode 100644 index 0000000..52eef65 --- /dev/null +++ b/pages/ena-submission.md @@ -0,0 +1,92 @@ +--- +title: ENA data submission toolbox +contributors: [Bert Droesbeke, Flora D'Anna] +description: High scale publishing of data to ENA using easy to use metadata templates. +training: + - name: Screencast of the Galaxy ENA upload tool + registry: YouTube + url: +# More information on how to fill in this metadata section can be found here https://www.infectious-diseases-toolkit.org/contribute/page-metadata +--- + + +## Introduction + +The (ENA) data submission toolbox simplifies the submission of sequence data, including raw reads and assembled sequences, for researchers. European Nucleotide Archive (ENA) is a fully open repository dedicated to storing raw sequencing data, assemblies, and annotation data. +Whether you're an experienced bioinformatician or a researcher with limited informatics background, our tools and workflows should help you in the submission process of both metadata and data to ENA. + +## Its components + +{% include image.html file="ena-submission-toolbox.svg" caption="Figure 1. Overview of the ENA submission toolbox components." %} + + +### ENA upload CLI + +Command line tool allowing submission of raw reads data and respective metadata to ENA using tabular files or an excel sheet. Programmatically submit study, sample, run and experiment objects without the need of logging in to the ENA Webin interface. [Documentation can be found here](https://github.com/usegalaxy-eu/ena-upload-cli) + +- Submission of raw reads data and metadata +- Using tabular files or an excel sheet to easily capture the metadata +- Add, modify, cancel and release study, experiment, run and sample objects +- Safe credential management using a credentials file +- Client side validation using ENA checklists (samples) and official ENA XSD files (run, experiment and study) +- Can the provided tsv/xlsx templates to fill in the metadata + +### Galaxy + +[Galaxy](https://galaxyproject.org/eu/) is an open-source platform for FAIR data analysis that enables users to use tools from various domains through its graphical web interface. +Following tool wrappers make Galaxy your one stop shop for data preprocessing, analysing and submitting. + +- **Galaxy ENA upload tool** + + Galaxy tool wrapper of the ENA-upload-cli. Gives the command line tool a graphic user interface and adds support for interactive submissions. + [Tool at IUC](https://github.com/galaxyproject/tools-iuc/tree/master/tools/ena_upload). + + - Graphical user interface making it easy to use + - Raw read submissions + - ENA-upload-cli wrapper + - Add and modify ENA objects + - User based credential management + - Possibility to set a brokering account + - Easy data upload/management + - Available @ useGalaxy.eu, .be and .au + +- **Galaxy assembly submission tool** + + Galaxy wrapper to submit consensus sequences to ENA in an interactive way. The tool has the [Webin-CLI](https://github.com/enasequence/webin-cli) script of ENA as core and supports all sample checklists. [More information](https://github.com/usegalaxy-be/galaxytools/tree/main/consensus_sequence_ena_galaxy). + + +### Metadata templates + +Tabular-format and xlsx spreadsheet metadata templates required to submit data to ENA using the ENA-upload-cli or GALAXY ENA upload tool. A GitHub Action is put in place to automatically keep these templates up to date with the ENA sample checklists.[Discover the templates here](https://github.com/ELIXIR-Belgium/ENA-metadata-templates) + +- GitHub repo hosting tsv and xlsx templates for every checklist +- GitHub Actions to keep up to date with ENA XSD/checklist files +- Versions in sync with the ones from ENA-upload-CLI + + + +### Docker deployment + +When you can not use the Galaxy instances @ useGalaxy.eu, .be and .au, possibly due to GDPR reasons, we also offer ready to use Docker containers. The Docker container is shipped with the previously mentioned Galaxy tools and deploys locally a fully usable Galaxy instance. +[More information on how to use the Docker solution](https://github.com/ELIXIR-Belgium/ena-upload-container). + +- Easy setup of a local Galaxy instance +- Developed during COVID-19 +- The Galaxy instance contains: + - Tool to easily clean raw reads from human reads + - ENA-upload tool + - ENA Assembly submission tool + - Workflows to perform SARS-CoV-2 analysis + + +## Using DataHub to manage your metadata + +- Research metadata management platform based on SEEK +- Leveraging the DataHub metadata templates +- Enforcing and promoting the ENA standards and checklists +- Existing ISA-JSON export function +- ISA-JSON as machine actionable metadata carrier + + +{% include image.html file="datahub-to-ena.svg" caption="Figure 2.Submission flow of metadata from DataHub to ENA" %} + From ac996990f08089174bf99f1637ad7444fb8e2416 Mon Sep 17 00:00:00 2001 From: bedroesb Date: Thu, 7 Mar 2024 19:39:53 +0100 Subject: [PATCH 02/12] extra line --- pages/ena-submission.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pages/ena-submission.md b/pages/ena-submission.md index 52eef65..560f89c 100644 --- a/pages/ena-submission.md +++ b/pages/ena-submission.md @@ -13,7 +13,8 @@ training: ## Introduction The (ENA) data submission toolbox simplifies the submission of sequence data, including raw reads and assembled sequences, for researchers. European Nucleotide Archive (ENA) is a fully open repository dedicated to storing raw sequencing data, assemblies, and annotation data. -Whether you're an experienced bioinformatician or a researcher with limited informatics background, our tools and workflows should help you in the submission process of both metadata and data to ENA. +The toolbox offers a single-step submission process, a graphical user interface, tabular-formatted metadata and client-side validation, for every sample checklist supported at ENA. + ## Its components From 7d5ef0bce5c97729d734fac735694562c110e742 Mon Sep 17 00:00:00 2001 From: bedroesb Date: Fri, 8 Mar 2024 10:23:43 +0100 Subject: [PATCH 03/12] better image --- images/datahub-to-ena.svg | 96 ++++----- images/ena-submission-toolbox.svg | 335 ++++++++++++++++++++---------- pages/ena-submission.md | 6 +- 3 files changed, 280 insertions(+), 157 deletions(-) diff --git a/images/datahub-to-ena.svg b/images/datahub-to-ena.svg index f44ada7..f6103ea 100644 --- a/images/datahub-to-ena.svg +++ b/images/datahub-to-ena.svg @@ -1,14 +1,14 @@ @@ -28,57 +28,58 @@ @@ -126,12 +127,13 @@ id="image18" /> @@ -152,7 +154,7 @@ id="path23" /> + id="defs53" /> + id="g53" + transform="translate(-25.973753,-50.429994)"> - + id="path12" /> + id="path13" /> + id="path14" /> + id="path15" /> + id="path16" /> + id="path17" /> + id="path18" /> + id="path19" /> + + + + + + + + + + id="path27" /> + id="path28" /> + + + + + id="path33" /> + + id="path35" /> + id="path36" /> + id="path37" /> - - - - - - + id="path38" /> + + + id="path41" /> + + + + + + + + + id="path50" /> + id="path51" /> + + diff --git a/pages/ena-submission.md b/pages/ena-submission.md index 560f89c..cff5442 100644 --- a/pages/ena-submission.md +++ b/pages/ena-submission.md @@ -6,14 +6,12 @@ training: - name: Screencast of the Galaxy ENA upload tool registry: YouTube url: -# More information on how to fill in this metadata section can be found here https://www.infectious-diseases-toolkit.org/contribute/page-metadata --- ## Introduction -The (ENA) data submission toolbox simplifies the submission of sequence data, including raw reads and assembled sequences, for researchers. European Nucleotide Archive (ENA) is a fully open repository dedicated to storing raw sequencing data, assemblies, and annotation data. -The toolbox offers a single-step submission process, a graphical user interface, tabular-formatted metadata and client-side validation, for every sample checklist supported at ENA. +European Nucleotide Archive (ENA) is a fully open repository dedicated to storing raw sequencing data, assemblies, and annotation data. The ENA data submission toolbox simplifies the submission of sequence data, including raw reads and assembled sequences by offering a single-step submission process, a graphical user interface, tabular-formatted metadata and client-side validation, for every sample checklist supported at ENA. ## Its components @@ -89,5 +87,5 @@ When you can not use the Galaxy instances @ useGalaxy.eu, .be and .au, possibly - ISA-JSON as machine actionable metadata carrier -{% include image.html file="datahub-to-ena.svg" caption="Figure 2.Submission flow of metadata from DataHub to ENA" %} +{% include image.html file="datahub-to-ena.svg" caption="Figure 2. Submission flow of metadata from DataHub to ENA" %} From 11b730445eb81a6d9eccb1f3e05623ce1efe657a Mon Sep 17 00:00:00 2001 From: bedroesb Date: Fri, 8 Mar 2024 11:05:12 +0100 Subject: [PATCH 04/12] adding buttons --- pages/ena-submission.md | 72 +++++++++++++++++++++++++++++------------ 1 file changed, 52 insertions(+), 20 deletions(-) diff --git a/pages/ena-submission.md b/pages/ena-submission.md index cff5442..6816346 100644 --- a/pages/ena-submission.md +++ b/pages/ena-submission.md @@ -6,6 +6,9 @@ training: - name: Screencast of the Galaxy ENA upload tool registry: YouTube url: + - name: Galaxy Training network + registry: + url: --- @@ -21,8 +24,9 @@ European Nucleotide Archive (ENA) is a fully open repository dedicated to storin ### ENA upload CLI -Command line tool allowing submission of raw reads data and respective metadata to ENA using tabular files or an excel sheet. Programmatically submit study, sample, run and experiment objects without the need of logging in to the ENA Webin interface. [Documentation can be found here](https://github.com/usegalaxy-eu/ena-upload-cli) +Command line tool allowing submission of raw reads data and respective metadata to ENA using tabular files or an excel sheet. Programmatically submit study, sample, run and experiment objects without the need of logging in to the ENA Webin interface. +Key features: - Submission of raw reads data and metadata - Using tabular files or an excel sheet to easily capture the metadata - Add, modify, cancel and release study, experiment, run and sample objects @@ -30,62 +34,90 @@ Command line tool allowing submission of raw reads data and respective metadata - Client side validation using ENA checklists (samples) and official ENA XSD files (run, experiment and study) - Can the provided tsv/xlsx templates to fill in the metadata +[Github](https://github.com/usegalaxy-eu/ena-upload-cli){: class="btn btn-primary rounded-pill"} +[Install using pip](https://pypi.org/project/ena-upload-cli/){: class="btn btn-primary rounded-pill"} + ### Galaxy [Galaxy](https://galaxyproject.org/eu/) is an open-source platform for FAIR data analysis that enables users to use tools from various domains through its graphical web interface. -Following tool wrappers make Galaxy your one stop shop for data preprocessing, analysing and submitting. +Following tool wrappers make Galaxy your one stop shop for data preprocessing, analysing and submitting. Both tools can be installed through the Galaxy toolshed. Ask the administrator of you Galaxy instance if this is not yet the case. + +#### Galaxy ENA upload tool + +Galaxy tool wrapper of the ENA-upload-cli. Gives the command line tool a graphic user interface and adds support for interactive submissions. + +Key features: +- Graphical user interface making it easy to use +- Raw read submissions +- ENA-upload-cli wrapper +- Add and modify ENA objects +- User based credential management +- Possibility to set a brokering account +- Easy data upload/management +- Available @ useGalaxy.eu, .be and .au + +[Use at useGalaxy.be](https://usegalaxy.be/?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fiuc%2Fena_upload%2Fena_upload){: class="btn btn-primary rounded-pill"} +[Github](https://github.com/galaxyproject/tools-iuc/tree/master/tools/ena_upload){: class="btn btn-primary rounded-pill"} + -- **Galaxy ENA upload tool** - Galaxy tool wrapper of the ENA-upload-cli. Gives the command line tool a graphic user interface and adds support for interactive submissions. - [Tool at IUC](https://github.com/galaxyproject/tools-iuc/tree/master/tools/ena_upload). +#### Galaxy assembly submission tool - - Graphical user interface making it easy to use - - Raw read submissions - - ENA-upload-cli wrapper - - Add and modify ENA objects - - User based credential management - - Possibility to set a brokering account - - Easy data upload/management - - Available @ useGalaxy.eu, .be and .au +Galaxy wrapper to submit consensus sequences to ENA in an interactive way. The tool has the [Webin-CLI](https://github.com/enasequence/webin-cli) script of ENA as core and supports all sample checklists. -- **Galaxy assembly submission tool** +Key features: +- Interactive submission of the metadata +- Possibility to set a brokering account +- Easy data upload/management +- Available @ useGalaxy.be - Galaxy wrapper to submit consensus sequences to ENA in an interactive way. The tool has the [Webin-CLI](https://github.com/enasequence/webin-cli) script of ENA as core and supports all sample checklists. [More information](https://github.com/usegalaxy-be/galaxytools/tree/main/consensus_sequence_ena_galaxy). +[Use at useGalaxy.be](https://usegalaxy.be/?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fieguinoa%2Fena_webin_cli%2Fena_consensus_submit){: class="btn btn-primary rounded-pill"} +[Github](https://github.com/usegalaxy-be/galaxytools/tree/main/consensus_sequence_ena_galaxy){: class="btn btn-primary rounded-pill"} ### Metadata templates -Tabular-format and xlsx spreadsheet metadata templates required to submit data to ENA using the ENA-upload-cli or GALAXY ENA upload tool. A GitHub Action is put in place to automatically keep these templates up to date with the ENA sample checklists.[Discover the templates here](https://github.com/ELIXIR-Belgium/ENA-metadata-templates) +Tabular-format and xlsx spreadsheet metadata templates required to submit data to ENA using the ENA-upload-cli or GALAXY ENA upload tool. A GitHub Action is put in place to automatically keep these templates up to date with the ENA sample checklists. +Key features: - GitHub repo hosting tsv and xlsx templates for every checklist - GitHub Actions to keep up to date with ENA XSD/checklist files - Versions in sync with the ones from ENA-upload-CLI - +[Github](https://github.com/ELIXIR-Belgium/ENA-metadata-templates){: class="btn btn-primary rounded-pill"} ### Docker deployment When you can not use the Galaxy instances @ useGalaxy.eu, .be and .au, possibly due to GDPR reasons, we also offer ready to use Docker containers. The Docker container is shipped with the previously mentioned Galaxy tools and deploys locally a fully usable Galaxy instance. -[More information on how to use the Docker solution](https://github.com/ELIXIR-Belgium/ena-upload-container). -- Easy setup of a local Galaxy instance +Key features: +- Easy deployment of a local Galaxy instance - Developed during COVID-19 +- Data stays on-premise until submission - The Galaxy instance contains: - Tool to easily clean raw reads from human reads - ENA-upload tool - ENA Assembly submission tool - Workflows to perform SARS-CoV-2 analysis +[Github](https://github.com/ELIXIR-Belgium/ena-upload-container){: class="btn btn-primary rounded-pill"} ## Using DataHub to manage your metadata +{% include image.html file="datahub-to-ena.svg" caption="Figure 2. Submission flow of metadata from DataHub to ENA" %} + +DataHub is a free and open platform for easier Research metaData Management in Life Science. Based on SEEK, it has an improved way of handling samples and the possibility to load sample templates for the creation of the metadata structure. This functionality is ideal for the creation of ENA specific metadata templates which guarantee the compatibility with the repository when a researcher is using the platform. + +Key features: - Research metadata management platform based on SEEK - Leveraging the DataHub metadata templates - Enforcing and promoting the ENA standards and checklists - Existing ISA-JSON export function - ISA-JSON as machine actionable metadata carrier +[Try out on DataHub](https://datahub-test.elixir-belgium.org/){: class="btn btn-primary rounded-pill"} + + +## Publication -{% include image.html file="datahub-to-ena.svg" caption="Figure 2. Submission flow of metadata from DataHub to ENA" %} From ea3b87c91c35347e9469a783160f10bc31442f7d Mon Sep 17 00:00:00 2001 From: bedroesb Date: Fri, 8 Mar 2024 11:54:25 +0100 Subject: [PATCH 05/12] more information --- pages/ena-submission.md | 70 ++++++++++++++++++++++------------------- 1 file changed, 38 insertions(+), 32 deletions(-) diff --git a/pages/ena-submission.md b/pages/ena-submission.md index 6816346..db5b576 100644 --- a/pages/ena-submission.md +++ b/pages/ena-submission.md @@ -3,18 +3,18 @@ title: ENA data submission toolbox contributors: [Bert Droesbeke, Flora D'Anna] description: High scale publishing of data to ENA using easy to use metadata templates. training: - - name: Screencast of the Galaxy ENA upload tool + - name: Submission of SARS-CoV-2 sequence data to ENA using Galaxy registry: YouTube - url: - - name: Galaxy Training network + url: https://www.youtube.com/watch?v=POiQG-7O7rw + - name: "Galaxy Training network: Submitting sequence data to ENA" registry: - url: + url: https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/upload-data-to-ena/tutorial.html --- ## Introduction -European Nucleotide Archive (ENA) is a fully open repository dedicated to storing raw sequencing data, assemblies, and annotation data. The ENA data submission toolbox simplifies the submission of sequence data, including raw reads and assembled sequences by offering a single-step submission process, a graphical user interface, tabular-formatted metadata and client-side validation, for every sample checklist supported at ENA. +[European Nucleotide Archive (ENA)](https://www.ebi.ac.uk/ena/browser/) is a fully open repository dedicated to storing raw sequencing data, assemblies, and annotation data. The ENA data submission toolbox simplifies the submission of sequence data, including raw reads and assembled sequences by offering a single-step submission process, a graphical user interface, tabular-formatted metadata and client-side validation, for every sample checklist supported at ENA. ## Its components @@ -27,52 +27,56 @@ European Nucleotide Archive (ENA) is a fully open repository dedicated to storin Command line tool allowing submission of raw reads data and respective metadata to ENA using tabular files or an excel sheet. Programmatically submit study, sample, run and experiment objects without the need of logging in to the ENA Webin interface. Key features: -- Submission of raw reads data and metadata -- Using tabular files or an excel sheet to easily capture the metadata -- Add, modify, cancel and release study, experiment, run and sample objects +- Submit raw sequencing data and metadata +- High volume submissions +- Support for all sample types +- Use tabular files or an excel sheet to easily capture the metadata +- Add, modify, cancel and release study, experiment, run and sample objects taking away the need - Safe credential management using a credentials file - Client side validation using ENA checklists (samples) and official ENA XSD files (run, experiment and study) -- Can the provided tsv/xlsx templates to fill in the metadata +- Compatible with the provided tsv/xlsx templates to fill in the metadata (see below) -[Github](https://github.com/usegalaxy-eu/ena-upload-cli){: class="btn btn-primary rounded-pill"} -[Install using pip](https://pypi.org/project/ena-upload-cli/){: class="btn btn-primary rounded-pill"} +[Documentation + source](https://github.com/usegalaxy-eu/ena-upload-cli){: class="btn btn-light border rounded-pill"} +[Install using pip](https://pypi.org/project/ena-upload-cli/){: class="btn btn-light border rounded-pill"} ### Galaxy -[Galaxy](https://galaxyproject.org/eu/) is an open-source platform for FAIR data analysis that enables users to use tools from various domains through its graphical web interface. -Following tool wrappers make Galaxy your one stop shop for data preprocessing, analysing and submitting. Both tools can be installed through the Galaxy toolshed. Ask the administrator of you Galaxy instance if this is not yet the case. +[Galaxy](https://galaxyproject.org/eu/) is an open-source platform for FAIR data analysis that enables users to use tools from various domains through its graphical web interface.Following tool wrappers make Galaxy your one stop shop for data preprocessing, analysing and submitting. Both tools can be installed through the Galaxy toolshed. Ask the administrator of you Galaxy instance if this is not yet the case. #### Galaxy ENA upload tool -Galaxy tool wrapper of the ENA-upload-cli. Gives the command line tool a graphic user interface and adds support for interactive submissions. +This isn the Galaxy tool wrapper of the ENA-upload-cli mentioned above. The integration with Galaxy gives the command line tool a graphic user interface and adds support for interactive submissions. Key features: - Graphical user interface making it easy to use - Raw read submissions -- ENA-upload-cli wrapper +- ENA-upload-cli at its core - Add and modify ENA objects - User based credential management -- Possibility to set a brokering account +- Possibility to set a system-wide brokering account - Easy data upload/management -- Available @ useGalaxy.eu, .be and .au - -[Use at useGalaxy.be](https://usegalaxy.be/?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fiuc%2Fena_upload%2Fena_upload){: class="btn btn-primary rounded-pill"} -[Github](https://github.com/galaxyproject/tools-iuc/tree/master/tools/ena_upload){: class="btn btn-primary rounded-pill"} +- Available at [useGalaxy Europe](https://usegalaxy.eu/), [useGalaxy Belgium](https://usegalaxy.be/) and [useGalaxy Australia](https://usegalaxy.org.au/) +[Use at useGalaxy.be](https://usegalaxy.be/?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fiuc%2Fena_upload%2Fena_upload){: class="btn btn-light border rounded-pill"} +[Source](https://github.com/galaxyproject/tools-iuc/tree/master/tools/ena_upload){: class="btn btn-light border rounded-pill"} +[Tutorial on GTN](https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/upload-data-to-ena/tutorial.html#submitting-raw-sequence-data-reads-to-the-ena){: class="btn btn-light border rounded-pill"} +[Galaxy toolshed](https://toolshed.g2.bx.psu.edu/repository?repository_id=0db04aa13ef9d2f8){: class="btn btn-light border rounded-pill"} #### Galaxy assembly submission tool -Galaxy wrapper to submit consensus sequences to ENA in an interactive way. The tool has the [Webin-CLI](https://github.com/enasequence/webin-cli) script of ENA as core and supports all sample checklists. +Galaxy wrapper to submit consensus sequences to ENA in an interactive way. The tool has the [Webin-CLI](https://github.com/enasequence/webin-cli) script of ENA at its core and supports all sample checklists. Key features: - Interactive submission of the metadata - Possibility to set a brokering account - Easy data upload/management -- Available @ useGalaxy.be +- Available at [useGalaxy Belgium](https://usegalaxy.be/) -[Use at useGalaxy.be](https://usegalaxy.be/?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fieguinoa%2Fena_webin_cli%2Fena_consensus_submit){: class="btn btn-primary rounded-pill"} -[Github](https://github.com/usegalaxy-be/galaxytools/tree/main/consensus_sequence_ena_galaxy){: class="btn btn-primary rounded-pill"} +[Use at useGalaxy.be](https://usegalaxy.be/?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fieguinoa%2Fena_webin_cli%2Fena_consensus_submit){: class="btn btn-light border rounded-pill"} +[Source](https://github.com/usegalaxy-be/galaxytools/tree/main/consensus_sequence_ena_galaxy){: class="btn btn-light border rounded-pill"} +[Tutorial on GTN](https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/upload-data-to-ena/tutorial.html#submitting-consensus-sequences-to-ena){: class="btn btn-light border rounded-pill"} +[Galaxy toolshed](https://toolshed.g2.bx.psu.edu/repository?repository_id=dfa4f0fc31027b52){: class="btn btn-light border rounded-pill"} ### Metadata templates @@ -81,26 +85,27 @@ Tabular-format and xlsx spreadsheet metadata templates required to submit data t Key features: - GitHub repo hosting tsv and xlsx templates for every checklist -- GitHub Actions to keep up to date with ENA XSD/checklist files +- GitHub Actions to keep up to date with ENA XSD/checklist files to guarantee compatibility - Versions in sync with the ones from ENA-upload-CLI -[Github](https://github.com/ELIXIR-Belgium/ENA-metadata-templates){: class="btn btn-primary rounded-pill"} +[Documentation + source](https://github.com/ELIXIR-Belgium/ENA-metadata-templates){: class="btn btn-light border rounded-pill"} ### Docker deployment -When you can not use the Galaxy instances @ useGalaxy.eu, .be and .au, possibly due to GDPR reasons, we also offer ready to use Docker containers. The Docker container is shipped with the previously mentioned Galaxy tools and deploys locally a fully usable Galaxy instance. +When you can not use the Galaxy instances at useGalaxy.eu, .be and .au, possibly due to GDPR reasons, we also offer a ready to use Docker container. The Docker container is shipped with the previously mentioned Galaxy tools and deploys locally a fully usable Galaxy instance. Key features: - Easy deployment of a local Galaxy instance -- Developed during COVID-19 - Data stays on-premise until submission -- The Galaxy instance contains: - - Tool to easily clean raw reads from human reads +- Contains: - ENA-upload tool - ENA Assembly submission tool + - Tool to easily clean raw reads from human reads - Workflows to perform SARS-CoV-2 analysis -[Github](https://github.com/ELIXIR-Belgium/ena-upload-container){: class="btn btn-primary rounded-pill"} +[Documentation + source](https://github.com/ELIXIR-Belgium/ena-upload-container){: class="btn btn-light border rounded-pill"} +[Download image from Quay.io](https://quay.io/repository/galaxy/ena-upload){: class="btn btn-light border rounded-pill"} + ## Using DataHub to manage your metadata @@ -115,9 +120,10 @@ Key features: - Existing ISA-JSON export function - ISA-JSON as machine actionable metadata carrier -[Try out on DataHub](https://datahub-test.elixir-belgium.org/){: class="btn btn-primary rounded-pill"} +[Try out on DataHub](https://datahub-test.elixir-belgium.org/){: class="btn btn-light border rounded-pill"} ## Publication +Roncoroni, M., Droesbeke, B., Eguinoa, I., De Ruyck, K., D’Anna, F., Yusuf, D., Grüning, B., Backofen, R., & Coppens, F. (2021). A SARS-CoV-2 sequence submission tool for the European Nucleotide Archive. Bioinformatics, 37(21), 3983–3985. [https://doi.org/10.1093/bioinformatics/btab421](https://doi.org/10.1093/bioinformatics/btab421) From 53ba5bcc4a55fde2789fc5f098571f45f29cd65b Mon Sep 17 00:00:00 2001 From: bedroesb Date: Fri, 8 Mar 2024 11:55:55 +0100 Subject: [PATCH 06/12] adding urls --- pages/ena-submission.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/ena-submission.md b/pages/ena-submission.md index db5b576..9566743 100644 --- a/pages/ena-submission.md +++ b/pages/ena-submission.md @@ -111,7 +111,7 @@ Key features: {% include image.html file="datahub-to-ena.svg" caption="Figure 2. Submission flow of metadata from DataHub to ENA" %} -DataHub is a free and open platform for easier Research metaData Management in Life Science. Based on SEEK, it has an improved way of handling samples and the possibility to load sample templates for the creation of the metadata structure. This functionality is ideal for the creation of ENA specific metadata templates which guarantee the compatibility with the repository when a researcher is using the platform. +[DataHub](https://datahub-test.elixir-belgium.org/) is a free and open platform for easier Research metaData Management in Life Science. Based on [FAIRDOM SEEK](https://seek4science.org/), it has an improved way of handling samples and the possibility to load sample templates for the creation of the metadata structure. This functionality is ideal for the creation of ENA specific metadata templates which guarantee the compatibility with the repository when a researcher is using the platform. Key features: - Research metadata management platform based on SEEK From b9e282dd2749bbe16be970716fed21dd4877b42d Mon Sep 17 00:00:00 2001 From: bedroesb Date: Fri, 8 Mar 2024 11:56:41 +0100 Subject: [PATCH 07/12] typo --- pages/ena-submission.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/ena-submission.md b/pages/ena-submission.md index 9566743..09984f3 100644 --- a/pages/ena-submission.md +++ b/pages/ena-submission.md @@ -117,7 +117,7 @@ Key features: - Research metadata management platform based on SEEK - Leveraging the DataHub metadata templates - Enforcing and promoting the ENA standards and checklists -- Existing ISA-JSON export function +- Uses the existing ISA-JSON export function - ISA-JSON as machine actionable metadata carrier [Try out on DataHub](https://datahub-test.elixir-belgium.org/){: class="btn btn-light border rounded-pill"} From 0e15fd822b77d772b5df9620cfb2382b2071ee95 Mon Sep 17 00:00:00 2001 From: bedroesb Date: Fri, 8 Mar 2024 12:01:31 +0100 Subject: [PATCH 08/12] add more urls --- pages/ena-submission.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pages/ena-submission.md b/pages/ena-submission.md index 09984f3..9df0e21 100644 --- a/pages/ena-submission.md +++ b/pages/ena-submission.md @@ -24,7 +24,7 @@ training: ### ENA upload CLI -Command line tool allowing submission of raw reads data and respective metadata to ENA using tabular files or an excel sheet. Programmatically submit study, sample, run and experiment objects without the need of logging in to the ENA Webin interface. +Command line tool allowing submission of raw reads data and respective metadata to ENA using tabular files or an excel sheet. Programmatically submit study, sample, run and experiment objects without the need of logging in to the [ENA Webin interface](https://www.ebi.ac.uk/ena/submit/webin/login). Key features: - Submit raw sequencing data and metadata @@ -34,7 +34,7 @@ Key features: - Add, modify, cancel and release study, experiment, run and sample objects taking away the need - Safe credential management using a credentials file - Client side validation using ENA checklists (samples) and official ENA XSD files (run, experiment and study) -- Compatible with the provided tsv/xlsx templates to fill in the metadata (see below) +- Compatible with the provided tsv/xlsx templates to fill in the metadata ([see below](#metadata-templates)) [Documentation + source](https://github.com/usegalaxy-eu/ena-upload-cli){: class="btn btn-light border rounded-pill"} [Install using pip](https://pypi.org/project/ena-upload-cli/){: class="btn btn-light border rounded-pill"} From 59e66f4c640c371c5b7cc1047d474650250992f9 Mon Sep 17 00:00:00 2001 From: bedroesb Date: Fri, 8 Mar 2024 12:02:29 +0100 Subject: [PATCH 09/12] no sars cov 2 anymore --- pages/ena-submission.md | 1 - 1 file changed, 1 deletion(-) diff --git a/pages/ena-submission.md b/pages/ena-submission.md index 9df0e21..bfcf3c7 100644 --- a/pages/ena-submission.md +++ b/pages/ena-submission.md @@ -101,7 +101,6 @@ Key features: - ENA-upload tool - ENA Assembly submission tool - Tool to easily clean raw reads from human reads - - Workflows to perform SARS-CoV-2 analysis [Documentation + source](https://github.com/ELIXIR-Belgium/ena-upload-container){: class="btn btn-light border rounded-pill"} [Download image from Quay.io](https://quay.io/repository/galaxy/ena-upload){: class="btn btn-light border rounded-pill"} From de1d644536d0aca1afc7813a5de6234d63e8e0bd Mon Sep 17 00:00:00 2001 From: bedroesb Date: Fri, 8 Mar 2024 13:30:19 +0100 Subject: [PATCH 10/12] new image --- images/datahub-to-ena.svg | 101 ++++++++++++++++++++------------------ 1 file changed, 53 insertions(+), 48 deletions(-) diff --git a/images/datahub-to-ena.svg b/images/datahub-to-ena.svg index f6103ea..7fb01ef 100644 --- a/images/datahub-to-ena.svg +++ b/images/datahub-to-ena.svg @@ -1,14 +1,14 @@ @@ -24,7 +24,7 @@ + transform="translate(0,-85.57859)"> - - - + id="path28" /> + id="path29" /> + id="path30" /> + id="path31" /> + id="path32" /> + id="path33" /> + id="path34" /> + id="path35" /> + id="path36" /> + id="path37" /> + id="path38" /> + id="path39" /> + id="path40" /> + id="path41" /> + id="path42" /> + id="path43" /> + transform="matrix(0.30787454,0,0,0.32660709,331.48277,189.25956)" + id="g44"> + id="path44" /> + id="image44" /> + + + + From 79af0182aaf7443f1cd588f7f1ad2c7b31a284ef Mon Sep 17 00:00:00 2001 From: bedroesb Date: Fri, 8 Mar 2024 16:41:08 +0100 Subject: [PATCH 11/12] some improvements --- _data/sidebars/main.yml | 18 +++++++++--------- pages/ena-submission.md | 10 +++++----- pages/{covid-19 => }/faq.md | 3 +-- 3 files changed, 15 insertions(+), 16 deletions(-) rename pages/{covid-19 => }/faq.md (97%) diff --git a/_data/sidebars/main.yml b/_data/sidebars/main.yml index 3c3fe46..9928ef5 100644 --- a/_data/sidebars/main.yml +++ b/_data/sidebars/main.yml @@ -78,13 +78,13 @@ subitems: url: /arrayexpress - title: ENA data submission toolbox url: /ena-submission - - title: Covid-19 data submission - url: /covid-19/ subitems: - - title: SARS-CoV-2 raw reads submission in Galaxy - url: /covid-19/sarscov2_submission - - title: SARS-CoV-2 assembly submission in Galaxy - url: /covid-19/sarscov2_assembly_submission - - title: Frequently asked questions - url: /covid-19/faq - + - title: SARS-CoV-2 usecase + url: /covid-19/ + subitems: + - title: SARS-CoV-2 raw reads submission in Galaxy + url: /covid-19/sarscov2_submission + - title: SARS-CoV-2 assembly submission in Galaxy + url: /covid-19/sarscov2_assembly_submission + - title: FAQ for Docker deployment + url: /faq diff --git a/pages/ena-submission.md b/pages/ena-submission.md index bfcf3c7..eb07fd7 100644 --- a/pages/ena-submission.md +++ b/pages/ena-submission.md @@ -14,7 +14,7 @@ training: ## Introduction -[European Nucleotide Archive (ENA)](https://www.ebi.ac.uk/ena/browser/) is a fully open repository dedicated to storing raw sequencing data, assemblies, and annotation data. The ENA data submission toolbox simplifies the submission of sequence data, including raw reads and assembled sequences by offering a single-step submission process, a graphical user interface, tabular-formatted metadata and client-side validation, for every sample checklist supported at ENA. +The [European Nucleotide Archive (ENA)](https://www.ebi.ac.uk/ena/browser/) is a fully open repository dedicated to storing raw sequencing data, assemblies, and annotation data. The ENA data submission toolbox simplifies the submission of sequence data, including raw reads and assembled sequences by offering a single-step submission process, a graphical user interface, tabular-formatted metadata and client-side validation, for every sample checklist supported at ENA. ## Its components @@ -31,7 +31,7 @@ Key features: - High volume submissions - Support for all sample types - Use tabular files or an excel sheet to easily capture the metadata -- Add, modify, cancel and release study, experiment, run and sample objects taking away the need +- Add, modify, cancel and release ENA objects (study, experiment, run and sample) taking away the need login into ENA Webin. - Safe credential management using a credentials file - Client side validation using ENA checklists (samples) and official ENA XSD files (run, experiment and study) - Compatible with the provided tsv/xlsx templates to fill in the metadata ([see below](#metadata-templates)) @@ -41,11 +41,11 @@ Key features: ### Galaxy -[Galaxy](https://galaxyproject.org/eu/) is an open-source platform for FAIR data analysis that enables users to use tools from various domains through its graphical web interface.Following tool wrappers make Galaxy your one stop shop for data preprocessing, analysing and submitting. Both tools can be installed through the Galaxy toolshed. Ask the administrator of you Galaxy instance if this is not yet the case. +[Galaxy](https://galaxyproject.org/eu/) is an open-source platform for FAIR data analysis that enables users to use tools from various domains through its graphical web interface. The 2 tool wrappers listed below make Galaxy your one stop shop for data preprocessing, analysing and submitting. Both tools can be installed through the Galaxy toolshed. #### Galaxy ENA upload tool -This isn the Galaxy tool wrapper of the ENA-upload-cli mentioned above. The integration with Galaxy gives the command line tool a graphic user interface and adds support for interactive submissions. +This is the Galaxy tool wrapper of the ENA-upload-cli mentioned above. The integration with Galaxy gives the command line tool a graphic user interface and adds support for interactive submissions. Key features: - Graphical user interface making it easy to use @@ -110,7 +110,7 @@ Key features: {% include image.html file="datahub-to-ena.svg" caption="Figure 2. Submission flow of metadata from DataHub to ENA" %} -[DataHub](https://datahub-test.elixir-belgium.org/) is a free and open platform for easier Research metaData Management in Life Science. Based on [FAIRDOM SEEK](https://seek4science.org/), it has an improved way of handling samples and the possibility to load sample templates for the creation of the metadata structure. This functionality is ideal for the creation of ENA specific metadata templates which guarantee the compatibility with the repository when a researcher is using the platform. +[DataHub](https://datahub-test.elixir-belgium.org/) is a free and open platform for easier Research metaData Management in Life Sciences. Based on [FAIRDOM SEEK](https://seek4science.org/), it has an improved way of handling samples and the possibility to load sample templates for the creation of the metadata structure. This functionality is ideal for the creation of ENA specific metadata templates which guarantee the compatibility with the repository when using the platform. Key features: - Research metadata management platform based on SEEK diff --git a/pages/covid-19/faq.md b/pages/faq.md similarity index 97% rename from pages/covid-19/faq.md rename to pages/faq.md index a6bf04a..6d1174d 100644 --- a/pages/covid-19/faq.md +++ b/pages/faq.md @@ -1,6 +1,5 @@ --- -permalink: covid-19/faq -title: Frequently asked questions +title: FAQ for Docker deployment --- ### Why is the reads cleaning tool not showing files for input? From 92e1659b4ab851e4d1bc1cf2cd1920fc96c3919d Mon Sep 17 00:00:00 2001 From: floradanna <51486716+floradanna@users.noreply.github.com> Date: Mon, 11 Mar 2024 10:09:07 +0100 Subject: [PATCH 12/12] Update ena-submission.md review text about datahub --- pages/ena-submission.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/pages/ena-submission.md b/pages/ena-submission.md index eb07fd7..cd9c988 100644 --- a/pages/ena-submission.md +++ b/pages/ena-submission.md @@ -110,14 +110,16 @@ Key features: {% include image.html file="datahub-to-ena.svg" caption="Figure 2. Submission flow of metadata from DataHub to ENA" %} -[DataHub](https://datahub-test.elixir-belgium.org/) is a free and open platform for easier Research metaData Management in Life Sciences. Based on [FAIRDOM SEEK](https://seek4science.org/), it has an improved way of handling samples and the possibility to load sample templates for the creation of the metadata structure. This functionality is ideal for the creation of ENA specific metadata templates which guarantee the compatibility with the repository when using the platform. +[DataHub](https://datahub-test.elixir-belgium.org/) is a free and open platform for easier Research metaData Management in Life Sciences. Based on [FAIRDOM-SEEK](https://seek4science.org/), DataHub offers users the ability to effortlessly create sample metadata templates derived from ENA-specific templates for seamless compatibility with the ENA repository. Key features: -- Research metadata management platform based on SEEK -- Leveraging the DataHub metadata templates -- Enforcing and promoting the ENA standards and checklists -- Uses the existing ISA-JSON export function -- ISA-JSON as machine actionable metadata carrier +- Powered by FAIRDOM-SEEK open-source software, DataHub facilitates effective research metadata management +- Easily craft metadata templates tailored to various repositories +- Promoting and supporting ENA standards and checklists ensures data consistency and compliance +- Streamline data exchange with structured metadata export adhering to the ISA-JSON standard +- Promote ISA-JSON as a machine-actionable metadata carrier, enhancing interoperability + +{% include callout.html type="note" content="Please note that specific documentation on creating ISA-JSON for ENA in DataHub is a work in progress and will be available soon." %} [Try out on DataHub](https://datahub-test.elixir-belgium.org/){: class="btn btn-light border rounded-pill"}