Bump CPUs for fetch-and-ingest workflows #447
Conversation
I've been only bumping the memory but not the CPUs for the fetch-and-ingest workflows. Might as well use all the compute that we are paying for. GenBank should be using c5.9xlarge and GISAID should be using c5.12xlarge, so bumping CPUs to match the instances.¹ Maybe this will magically help #446? ¹ <https://aws.amazon.com/ec2/instance-types/c5/>
|
Last nights run finished under 12h 🎉 |
|
We could probably scrap those CPU limits altogether. All they do is make snakemake restrict the number of jobs run in parallel. This has obvious downsides and only rare benefits. The CPU scheduler figures out ways to give all jobs some share when oversubscribed. We don't really need snakemake to do pessimistic scheduling. |
For the aws batch runtime, the |
Yes, progress will still be made with oversubscription, but the run time of the whole workflow will increase, sometimes substantially, depending on the kind of workload. It's still better not to oversubscribe when you can avoid it. |
I've been only bumping the memory but not the CPUs for the fetch-and-ingest workflows. Might as well use all the compute that we are paying for. GenBank should be using c5.9xlarge and GISAID should be using c5.12xlarge, so bumping CPUs to match the instances.¹
Maybe this will magically help #446?
¹ https://aws.amazon.com/ec2/instance-types/c5/
Checklist