Manage File Assets for Batch Processing of a Genome

[josiahseaman]

In order to complete #10 Galleries of whole genomes, with Annotations #21, we need the ability to build up file assets and not recompute them for every job. In particular, this turned out to be onerous when creating Annotation FASTA files multiple time for each chromosome. Translocations caused one chromosome to need another chromosome's Annotation FASTA, which then created a lot of unnecessary work.

Faster Turn-Around batch

• Whole Batch goes in a folder with subfolders for each viz
• Read all contigs from memory, don’t use ungapped seq files
• Place each Annotation FASTA in the main folder: These are Assets!
  • Calculate ungapped annotation seq and hold in memory
• Gapped seq is computed, output, and dumped
• Do all file generation first, then only compute viz at the end
• Jobs are driven by asset generation
  • For (asset_name, job) check if asset_name exists, if not, run Job
  • If asset_name exists, read it into memory and move on to next job

Asset Order:

• Folder Structure
• Gapped FASTA (subfolder)
• Annotation FASTA (main folder)
• Gapped Annotation with differences (subfolder)
• PNG Composite
• DeepZoom and HTML
• Gallery HTML