Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for file staging inside a process #27

Open
adamjtaylor opened this issue Dec 21, 2023 · 0 comments
Open

Allow for file staging inside a process #27

adamjtaylor opened this issue Dec 21, 2023 · 0 comments

Comments

@adamjtaylor
Copy link
Collaborator

adamjtaylor commented Dec 21, 2023

In https://sagebionetworks.jira.com/browse/MC2DPQC-51 we found that files were being staged during the samplesheet_split step. As this occurs in the headjob rather than a process it proved impossible to slow/queue/limit this step and avoid hitting rate-limits on the GTex server.

Thanks to Brad for proposing a solution.

My feeling would be to add a new optional process called something like STAGE_FILES that occurs after SPLIT and before HISTOQC.

Passing through this process should be optional based on a param (say params.stage_in_process) which would be a bool with default false

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

1 participant