Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow ncov-ingest to upload to non-AWS remote file paths #308

Open
huddlej opened this issue May 13, 2022 · 0 comments
Open

Allow ncov-ingest to upload to non-AWS remote file paths #308

huddlej opened this issue May 13, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@huddlej
Copy link
Contributor

huddlej commented May 13, 2022

Context

Related to #307, we want to support users who run the ingest workflow on different cloud platforms like Google Cloud (for example, through Terra). Although we can bypass the ingest workflow's default upload target that expects an S3 bucket as its destination in WDL workflows on Terra and upload directly to GS from the WDL workflow, we would probably benefit in the long run from native support for uploads to other cloud platforms besides S3.

Description

As in #307, we should allow users to specify a generic "destination" URI corresponding to at least an S3 or GS bucket such that the upload rule uploads all data from the workflow to the given bucket. This means the remainder of the S3-specific logic in the workflow needs to be replaced with generic remote file support.

We might be able to reuse the remote file handling logic from ncov to support writing files to valid remote URIs. However, we may also want to compress files in the workflow prior to upload instead of compressing during the upload process.

@huddlej huddlej added the enhancement New feature or request label May 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
No open projects
Status: Prioritized
Development

No branches or pull requests

1 participant