diff --git a/addtnl_info/WG_internal.md b/addtnl_info/WG_internal.md index 1f688a7f..e0fd98eb 100644 --- a/addtnl_info/WG_internal.md +++ b/addtnl_info/WG_internal.md @@ -4,9 +4,13 @@ order: 997 # Working Groups and Internal Communications -Information regarding Network Working Groups and Internal Communications can be found on [HTAN's Synapse Wiki page](https://www.synapse.org/#!Synapse:syn17022193/wiki/584990). Access to the HTAN Wiki is restricted to HTAN Members. +Information regarding Network Working Groups and Internal Communications can be found on on HTAN's internal Synapse wiki pages: + +- **Phase 1 Centers**: [Synapse HTAN Phase 1 Wiki page](https://www.synapse.org/#!Synapse:syn17022193/wiki/584990). +- **Phase 2 Centers**: [Synapse HTAN Phase 2 Wiki page](https://www.synapse.org/Synapse:syn63296487/wiki/629655). \ +:warning: The Phase 2 wiki pages are new. Phase 2 Center members will be provided access to this wiki in the near future. !!! Note -The HTAN Synapse Wiki page is restricted to HTAN members. Please contact htandcc@ds.dfci.harvard.edu if you are a member of HTAN and need access to the wiki. +HTAN Synapse Wiki pages are restricted to HTAN members. Please contact htandcc@ds.dfci.harvard.edu if you are a member of HTAN and need access to the wiki. !!! diff --git a/addtnl_info/tool_protocol.md b/addtnl_info/tool_protocol.md index b76bfcd7..f8f4e30a 100644 --- a/addtnl_info/tool_protocol.md +++ b/addtnl_info/tool_protocol.md @@ -4,12 +4,15 @@ order: 1000 # Submitting Tools -Computational tools developed or used to support HTAN research projects can be added to the HTAN tool catalog by filling out the tool curation form available on [HTAN's Synapse Wiki page](https://www.synapse.org/#!Synapse:syn17022193/wiki/584990). +Computational tools developed or used to support HTAN research projects can be added to the HTAN tool catalog by filling out the tool curation form available on HTAN's internal Synapse wiki pages: +- **Phase 1 Centers**: [Synapse HTAN Phase 1 Wiki page](https://www.synapse.org/#!Synapse:syn17022193/wiki/584990). +- **Phase 2 Centers**: [Synapse HTAN Phase 2 Wiki page](https://www.synapse.org/Synapse:syn63296487/wiki/629655). \ +:warning: The Phase 2 wiki pages are new. Phase 2 Center members will be provided access to this wiki in the near future. !!! Note -The HTAN Synapse Wiki page is restricted to HTAN members. Please contact htandcc@ds.dfci.harvard.edu if you are a member of HTAN and need access to the wiki. +HTAN Synapse Wiki pages are restricted to HTAN members. Please contact htandcc@ds.dfci.harvard.edu if you are a member of HTAN and need access to the wiki. !!! diff --git a/data_submission/Information_New_Centers.md b/data_submission/Information_New_Centers.md index fb46bc6b..3d27d737 100644 --- a/data_submission/Information_New_Centers.md +++ b/data_submission/Information_New_Centers.md @@ -38,7 +38,7 @@ Step 4 : Review the [HTAN Checklist for Acceptance of Data](../data_submission/checklist.md). Step 5 -: Review your Center’s responsibilities for [Data De-identification](https://docs.humantumoratlas.org/data_submission/data_deidentification/), complete your Center’s Data De-Identification Plan and submit the completed document to your DCC Data Liaison. +: Review your Center’s responsibilities for [Data De-identification](https://docs.humantumoratlas.org/data_submission/data_deidentification/). Step 6 : Instruct all Center members who will need to contribute or view data to create a [Synapse account](https://accounts.synapse.org/register1?appId=synapse.org). diff --git a/data_submission/clin_biospec_assay.md b/data_submission/clin_biospec_assay.md index dc0f39c3..be5be241 100644 --- a/data_submission/clin_biospec_assay.md +++ b/data_submission/clin_biospec_assay.md @@ -5,79 +5,84 @@ order: 994 # Submitting Assay Data and Metadata As stated in [Data Submission Introduction](../data_submission/overview.md), data submission involves two key steps: + 1. Uploading assay data files to Synapse; and 2. Completing and validating metadata using the Data Curator App (DCA). -!!! Once assay data files are submitted to Synapse, the files will have entityIDs (e.g. syn12345670) assigned to them. These can then be prepopulated into the manifests on the DCA. For this reason, assay files should be submitted before generating the associated manifests. -!!! +This page provides details regarding those steps. -This page provides details regarding those steps. Please note that the manual currently reflects the data submission process used in HTAN Phase 1. Changes may be implemented for HTAN Phase 2. +!!! +**Please note that the manual currently reflects the data submission process used in HTAN Phase 1. Changes may be implemented for HTAN Phase 2.** +!!! ![HTAN Data Submission Process](../img/Data_submission.svg) -To submit data, you will also need to understand the HTAN data model and specific requirements for your particular data type. For a general overview of the HTAN data model, please see [HTAN Data Model](../data_model/overview.md). To understand specific requirements for your data type, please see [Data Standards](https://humantumoratlas.org/standards). - -HTAN uses the Synapse [Portal](https://www.synapse.org) and [DCA](https://dca.app.sagebionetworks.org/), developed and maintained by [Sage Bionetworks](https://sagebionetworks.org/), to manage clinical, biospecimen and assay data submissions (dataset ingress). In order to submit data, your center should: - -1. [Have at least one user with Certified User status on Synapse.](#have-at-least-one-user-with-certified-user-status-on-synapse) -2. [Contact your Data Liaison to set up your project and cloud bucket.](#contact-your-data-liaison-to-set-up-your-project-and-cloud-bucket) -3. [Ensure the assay dataset conforms to the HTAN Data Model, uses HTAN Identifiers and does not contain Protected Health Information (PHI).](#ensure-the-dataset-conforms-to-the-htan-data-model-uses-htan-identifiers-and-does-not-contain-phi) -4. [Organize and upload your dataset to the Synapse Project](#organize-and-upload-your-dataset-to-the-synapse-project) -5. [Validate and submit metadata using the DCA.](#validate-and-submit-metadata-using-synapses-data-curator-app-dca) +## Data Submission Steps +1. [Complete Pre-submission Tasks](#pre-submission-tasks) +2. [Submit Data Files](#submit-data-files) +4. [Submit metadata](#submit-metadata) Please read the rest of this page for more information about each of these steps. -## Have at least one user with Certified User status on Synapse. -To upload files to the Synapse Platform, you need to be a [Synapse Certified User](https://help.synapse.org/docs/Synapse-User-Account-Types.2007072795.html). Because Synapse stores data from human subjects research, Sage Bionetworks requires that you demonstrate understanding of and compliance with privacy and security issues. You can complete your certification by taking a short certification quiz. Please see the Synapse [Certified User Documentation](https://help.synapse.org/docs/Synapse-User-Account-Types.2007072795.html) for more information. +### Pre-submission Tasks -## Contact your Data Liaison to set up your project and cloud bucket. +- [ ] Have at least one user with Certified User status on Synapse. -When you are ready to upload data, please contact your [data liaison](../data_submission/Data_Liaisons.md). Your data liaison will need to know: -1. Your centers -2. Who on your team will be doing the data upload. -3. The synapse usernames for team members identified in #2. +> To upload files to the Synapse Platform, you need to be a [Synapse Certified User](https://help.synapse.org/docs/Synapse-User-Account-Types.2007072795.html). You can complete your certification by taking a short certification quiz. Please see the Synapse [Certified User Documentation](https://help.synapse.org/docs/Synapse-User-Account-Types.2007072795.html) for more information. -Please have users obtain certified user status prior to contacting your data liaison. +- [ ] Contact your Data Liaison -With the above information, the DCC will initialize your Synapse project for metadata submission and a cloud storage location for dataset uploads. If the data submission is for a new atlas, the DCC will also create an HTAN atlas ID. Once your Synapse project has been initialized, your data liaison will reach out to you with the location of your Synapse project and you can begin uploading your data. +> When you are ready to upload data, please contact your [data liaison](../data_submission/Data_Liaisons.md). Please have users obtain certified user status prior to contacting your data liaison. -## Ensure the dataset conforms to the HTAN Data Model, uses HTAN Identifiers and does not contain PHI. +- [ ] Ensure the dataset conforms to the HTAN Data Model and uses HTAN Identifiers. -The HTAN Data Model is built upon data standards described on the [Data Standards](https://data.humantumoratlas.org/standards) page. All HTAN Centers are required to encode their clinical, biospecimen and assay data and metadata using the HTAN Data Model. If you have a new data type which is not currently represented in the HTAN Data Model, please contact your data liaison. +> The HTAN Data Model is built upon data standards described on the [Data Standards](https://data.humantumoratlas.org/standards) page. All HTAN Centers are required to encode their clinical, biospecimen and assay data and metadata using the HTAN Data Model. If you have a new data type which is not currently represented in the HTAN Data Model, please contact your data liaison. -A concrete way to understand the expectations for data submissions is to view the metadata templates (manifests) for clinical, biospecimen and assay data available in the ([DCA](https://dca.app.sagebionetworks.org/)). For any given dataset, you may be submitting: +> All data should be identified using HTAN identifiers. Please see the [Identifiers](../data_model/identifiers.md) and [Creating HTAN Identifiers](../data_submission/creating_ids.md) sections of this manual for more information regarding HTAN identifiers. -- clinical manifest(s), e.g. Demographics, Diagnosis -- biospecimen manifest(s) -- assay manifest(s), e.g. Bulk RNA-seq level 1 -- assay data files +- [ ] Ensure that your data does not contain PHI. -The first three items will be validated and submitted using the DCA. The last item, assay data files, only needs to be uploaded to the synapse project itself. +> **Please review your data to ensure that it does not contain PHI.** The HTAN DCC cannot accept data with PHI, including dates less than a year. For example, dates in metadata must be converted to days from an [index date](../data_submission/dates.md) and all image files must have PHI removed from file headers. -All data should be identified using HTAN identifiers. Please see the [HTAN Identifier](../data_model/identifiers.md) section of this manual for more information regarding HTAN identifiers. - -!!! *Please review your data to ensure that it does not contain PHI.* -!!! +### Submit Data Files -## Organize and upload your dataset to the Synapse Project +Organize your data using the **flattened data layout** described in Synapse's [Data Ingress Docs](https://dca-docs.scrollhelp.site/DCA/Working-version/HTAN/organize-your-data-upload#OrganizeyourDataUpload-FlattenedDataLayoutExample) -Please organize your data using the flattened data layout described in Synapse's [Data Ingress Docs](https://dca-docs.scrollhelp.site/DCA/Working-version/HTAN/organize-your-data-upload#OrganizeyourDataUpload-FlattenedDataLayoutExample) +Data files can be transferred using the Synapse User Interface (Synapse UI) or programmatically. -Data files can be transferred using the Synapse User Interface (Synapse UI) or programmatically. Please see Synapse's [Data Ingress Docs](https://dca-docs.scrollhelp.site/DCA/Working-version/HTAN/uploading-data) for more information on how to upload files. +- To upload files using the Synapse User Interface, follow Synapse's [Quick Overview: Uploading a File (via Synapse UI)](https://dca-docs.scrollhelp.site/DCA/Working-version/HTAN/uploading-data#UploadData-QuickOverview:UploadingaFile(viatheSynapseUI)) directions. +- To upload the files programmatically, please follow Synapse's directions to [Upload Data Using the Command Line](https://dca-docs.scrollhelp.site/DCA/Working-version/HTAN/uploading-data#UploadData-UploadDataUsingtheCommandLine). This involves using Synapse Client. !!! If you upload files to Synapse programmatically, please use synapseclient version 3.0.0 or higher. !!! -## Validate and submit metadata using Synapse's Data Curator App (DCA). +## Submit Metadata + +!!! +1. Only the DCA should be used to obtain manifests (metadata templates). +2. Always upload data files to Synapse first before submitting assay metadata. +!!! The DCA contains HTAN-specific manifests (metadata templates) which can be 1. completed on the app, or 2. downloaded, completed and uploaded back to the DCA. -Manifests for assay data will be pre-populated with assay file entityIDs once they are associated with a particular Synapse dataset folder. Once the manifests are completed by your center, they should then be validated and submitted via the DCA. DCA validation checks for a subset of common errors. If any of these errors are found, you can edit the metadata and then revalidate and submit. +Manifests for assay data will be pre-populated with assay file entityIDs once they are associated with a particular Synapse dataset folder. + +Once the manifests are completed by your center, they should then be validated and submitted via the DCA. The DCA will perform validation checks for a subset of common errors. If any of these errors are found, you can edit the metadata, revalidate and submit. !!! Please note: If you have added assay files to a Synapse folder where there is a pre-existing manifest or you are adding records to a pre-existing clinical data or biospecimen manifest, **please update the existing manifest on the DCA app or download the existing manifest from the DCA** to make updates. **Do not use a local copy of the manifest at your center to make updates**. Local copies may be out of sync with the data in Synapse. !!! Please see Synapse's [Data Ingress Docs](https://dca-docs.scrollhelp.site/DCA/Working-version/HTAN/validate-and-submit-your-metadata) for more details regarding the web app. + +## Useful Links and Guides + +### Synapse and the DCA +- Synapse [Portal](https://www.synapse.org) +- [DCA](https://dca.app.sagebionetworks.org/), developed and maintained by [Sage Bionetworks](https://sagebionetworks.org/) + +### Understanding the HTAN Data Model +- To understand the general structure of the HTAN Data Model and HTAN Identifiers, please see the [HTAN Data Model](../data_model/overview.md) section of this manual. +- To understand the Data Model Manifests/Metadata Attributes, please see the [Data Standards](https://humantumoratlas.org/standards) section of the HTAN Portal. There, you can download manifest summaries. These **cannot be used for metadata submission**, but can help you prepare your metadata. \ No newline at end of file diff --git a/img/Data_submission.svg b/img/Data_submission.svg index 6374b85e..2f00e30a 100644 --- a/img/Data_submission.svg +++ b/img/Data_submission.svg @@ -1 +1 @@ - \ No newline at end of file + \ No newline at end of file