Merge pull request #3 from BD2KGenomics/feature/runcmdtryretrydelay

Add try except, retry, and delay to new function for running a command
BD2KGenomics · Feb 19, 2017 · dd49a18 · dd49a18
2 parents 77951c7 + 8929740
commit dd49a18
Show file tree

Hide file tree

Showing 8 changed files with 211 additions and 180 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -22,7 +22,7 @@ RUN apt-get update && apt-get install --yes \
     build-essential
 
 RUN pip install --upgrade pip
-RUN pip install jsonschema jsonmerge openpyxl sets json-spec elasticsearch semver luigi
+RUN pip install jsonschema jsonmerge openpyxl sets json-spec elasticsearch semver
 
 #install cwltool in the container
 #use the version required by dockstore
@@ -87,4 +87,3 @@ ENV DOCKSTORE_ROOT 1
 #becuase ubuntu is not a member of the host's docker.sock docker group
 #and there is no way to set this up at build time
 USER root
-#USER ubuntu
diff --git a/Dockstore.cwl b/Dockstore.cwl
@@ -5,7 +5,7 @@ id: "dockstore-tool-runner"
 label: "A Dockstore tool that can call download from Redwood, call another Dockstore tool, and then upload back to Redwood."
 cwlVersion: v1.0
 doc: |
-    ![build_status](https://quay.io/ucsc_cgl/DockstoreRunner/status)
+    ![build_status](https://quay.io/repository/ucsc_cgl/dockstore-tool-runner/status)
     A Docker container from which Dockstore containers may be launched.
     ```
     Usage:
@@ -24,7 +24,7 @@ dct:creator:
 
 requirements:
   - class: DockerRequirement
-    dockerPull: "quay.io/ucsc_cgl/dockstore-tool-runner:1.0.7"
+    dockerPull: "quay.io/ucsc_cgl/dockstore-tool-runner:1.0.8"
 hints:
   - class: ResourceRequirement
     coresMin: 1

diff --git a/Dockstore.json b/Dockstore.json
@@ -1,11 +1,11 @@
 {
   "redwood_token": "<storage service token>",
-  "redwood_host": "storage2.ucsc-cgl.org",
-  "json_encoded": "ew0KCSJmYXN0cV9maWxlIjogW3sNCgkJImNsYXNzIjogIkZpbGUiLA0KCQkicGF0aCI6ICJyZWR3b29kOi8vc3RvcmFnZTIudWNzYy1jZ2wub3JnLzhlYmRiMDNhLTNjOTktNWYzMi04MTFjLTlkNzRiODgxNTFlYy8yZWFkY2M2NS00NGFmLTUyN2MtYTFhNy0yMmMzYTU1ZDczNmUvRVJSMDMwODg2XzEuZmFzdHEuZ3oiDQoJfSwgew0KCQkiY2xhc3MiOiAiRmlsZSIsDQoJCSJwYXRoIjogInJlZHdvb2Q6Ly9zdG9yYWdlMi51Y3NjLWNnbC5vcmcvOGViZGIwM2EtM2M5OS01ZjMyLTgxMWMtOWQ3NGI4ODE1MWVjLzgzNDUyM2YzLTdkZGYtNTA4Ni1hMTczLTEwODA2MGFlZWU3Ny9FUlIwMzA4ODZfMi5mYXN0cS5neiINCgl9XSwNCgkicmVwb3J0X2ZpbGVzIjogW3sNCgkJImNsYXNzIjogIkZpbGUiLA0KCQkicGF0aCI6ICIuL3RtcC9FUlIwMzA4ODZfMl9mYXN0cWMuaHRtbCINCgl9LCB7DQoJCSJjbGFzcyI6ICJGaWxlIiwNCgkJInBhdGgiOiAiLi90bXAvRVJSMDMwODg2XzFfZmFzdHFjLmh0bWwiDQoJfV0sDQoJInppcHBlZF9maWxlcyI6IFt7DQoJCSJjbGFzcyI6ICJGaWxlIiwNCgkJInBhdGgiOiAiLi90bXAvRVJSMDMwODg2XzJfZmFzdHFjLnppcCINCgl9LCB7DQoJCSJjbGFzcyI6ICJGaWxlIiwNCgkJInBhdGgiOiAiLi90bXAvRVJSMDMwODg2XzFfZmFzdHFjLnppcCINCgl9XQ0KfQ==",
-  "docker_uri": "quay.io/wshands/fastqc:latest",
-  "dockstore_url": "https://dockstore.org/containers/quay.io/wshands/fastqc",
+  "redwood_host": "storage.ucsc-cgl.org",
+  "json_encoded": "ew0KCSJmYXN0cV9maWxlcyI6IFt7DQoJCSJjbGFzcyI6ICJGaWxlIiwNCgkJInBhdGgiOiAicmVkd29vZDovL3N0b3JhZ2UudWNzYy1jZ2wub3JnL2ZiM2RkODVkLTM2N2UtNWI4ZC05OTI5LTk1MTY0MDg5ZDEwZi83M2YyMzYyNS04ZTU1LTU0MDgtOWY0ZS1hMmRlZDg0MGE2NWQvTkExMjg3OC1OR3YzLUxBQjEzNjAtQV8xLmZhc3RxLmd6Ig0KCX1dLA0KCSJ6aXBwZWRfZmlsZSI6IHsNCgkJImNsYXNzIjogIkZpbGUiLA0KCQkicGF0aCI6ICIvdG1wL2Zhc3RxY19yZXBvcnRzLnRhci5neiINCgl9DQp9",
+  "docker_uri": "quay.io/briandoconnor/fastqc:0.11.5",
+  "dockstore_url": "https://dockstore.org/containers/quay.io/briandoconnor/fastqc",
   "workflow_type": "sequence_upload_qc_report ",
-  "parent_uuids": "<parent uuid>",
+  "parent_uuids": "d2545c4e-dd7d-5a07-b598-e9acba87228f",
   "tmpdir": "<path in $TMPDIR env var>",
   "vm_instance_type": "m4.4xlarge",
   "vm_region": "us-west-2",

diff --git a/DockstoreRunner.py b/DockstoreRunner.py
diff --git a/README.md b/README.md
@@ -31,68 +31,87 @@ Now to setup:
 
     virtualenv env
     source env/bin/activate
-    pip install jsonschema jsonmerge openpyxl sets json-spec elasticsearch semver luigi python-dateutil cwl-runner cwltool==1.0.20160712154127 schema-salad==1.14.20160708181155 avro==1.8.1 typing
+    pip install jsonschema jsonmerge openpyxl sets json-spec elasticsearch semver luigi python-dateutil setuptools==28.8.0 cwl-runner cwltool==1.0.20160712154127 schema-salad==1.14.20160708181155 avro==1.8.1 typing
 
 Alternatively, you may want to use Conda, see [here](http://conda.pydata.org/docs/_downloads/conda-pip-virtualenv-translator.html)
  [here](http://conda.pydata.org/docs/test-drive.html), and [here](http://kylepurdon.com/blog/using-continuum-analytics-conda-as-a-replacement-for-virtualenv-pyenv-and-more.html)
  for more information.
 
     conda create -n dockstore-tool-runner-project python=2.7.11
     source activate dockstore-tool-runner-project
-    pip install jsonschema jsonmerge openpyxl sets json-spec elasticsearch semver luigi python-dateutil cwl-runner cwltool==1.0.20160316150250 schema-salad==1.7.20160316150109 avro==1.7.7 typing
+    pip install jsonschema jsonmerge openpyxl sets json-spec elasticsearch semver luigi python-dateutil setuptools==28.8.0 cwl-runner cwltool==1.0.20160712154127 schema-salad==1.14.20160708181155 avro==1.8.1 typing
 
 ### Patch CWLTools
 
 Unfortunately, we need to patch `cwltool` so we can properly handle calling nested Docker containers through it.  Specifically, we need to pass in the Docker socket and also ensure the working directory paths are consistent between the various layers of Docker calls.  If you have installed cwltool via pip in a virtualenv or conda environment make sure you patch that one and not the system version.  Customize the below for your environment.
 
-    patch -d /usr/local/lib/python2.7/dist-packages/cwltool/ < /usr/local/lib/python2.7/dist-packages/cwltool/job.patch
+    patch -d /usr/local/lib/python2.7/dist-packages/cwltool/ < job.patch
+
+### Install Dockstore CLI
+
+Take a look at http://dockstore.org and go through the onboarding process.
+
+That being said, this particular tool is designed to work with Dockstore CLI 1.0.1.  A copy is located in [dockstore](Dockstore/dockstore) so you can make sure you're using the version that matches the version of cwltool you patched above.
 
 ### Redwood Client
 
-You will need a copy of the Redwood client, you can download it from [here](https://s3-us-west-2.amazonaws.com/beni-dcc-storage-dev/ucsc-storage-client.tar.gz).
+You will need a copy of the Redwood client (assuming you are calling the tool outside of docker/dockstore), you can download it from [here](https://s3-us-west-2.amazonaws.com/beni-dcc-storage-dev/ucsc-storage-client.tar.gz).  You will need a token to upload/download data as well.  This assumes you're pulling/pushing to Redwood at UCSC.  If not, you don't need the client or token (instead, for example, you'll pull/push data to S3).
+
+### Make a Temp Dir
+
+So the dcc-dockstore-tool-runner is a Dockstore-based tool that calls another Dockstore-based tool.  This is a complex thing, since it means we need to 1) patch CWLtools to consistently pass in the Docker socket and 2) a common shared data path used by the nested containers.  The patch hardcodes this shared location to be `/datastore` so please create that before running these tools.  
+
+    sudo mkdir /datastore
+    sudo chmod a+rwx /datastore
+
+You should make sure /datastore is a large volume if you intend to work with large inputs/outputs.
+
+If you're on a mac, make sure you allow Docker to mount this otherwise you'll see an error like:
+
+![docs/mac_mount.png](docs/mac_mount.png)
+
+## Testing
+
+Lowest level to highest level.  The Dockstore CLI pointed to the released version of this tool is the way most users will call this.
 
-### Testing Command
+### Testing Python Command Directly
 
-The command below will download samples from Redwood, run fastqc from Dockstore on two fastq files, and then upload the results back to a Redwood storage system.
+The command below will download samples from Redwood, run fastqc from Dockstore on two fastq files, and then upload the results back to a Redwood storage system.  This is a real file and requires controlled access and a token to work.  You need to have cwltool installed/patched, Docker installed.  See above.
 
     # example with real files
-    python DockstoreRunner.py --redwood-path `pwd`/ucsc-storage-client --redwood-token `cat accessToken` --redwood-host storage2.ucsc-cgl.org --json-encoded ew0KCSJmYXN0cV9maWxlIjogW3sNCgkJImNsYXNzIjogIkZpbGUiLA0KCQkicGF0aCI6ICJyZWR3b29kOi8vc3RvcmFnZTIudWNzYy1jZ2wub3JnLzhlYmRiMDNhLTNjOTktNWYzMi04MTFjLTlkNzRiODgxNTFlYy8yZWFkY2M2NS00NGFmLTUyN2MtYTFhNy0yMmMzYTU1ZDczNmUvRVJSMDMwODg2XzEuZmFzdHEuZ3oiDQoJfSwgew0KCQkiY2xhc3MiOiAiRmlsZSIsDQoJCSJwYXRoIjogInJlZHdvb2Q6Ly9zdG9yYWdlMi51Y3NjLWNnbC5vcmcvOGViZGIwM2EtM2M5OS01ZjMyLTgxMWMtOWQ3NGI4ODE1MWVjLzgzNDUyM2YzLTdkZGYtNTA4Ni1hMTczLTEwODA2MGFlZWU3Ny9FUlIwMzA4ODZfMi5mYXN0cS5neiINCgl9XSwNCgkicmVwb3J0X2ZpbGVzIjogW3sNCgkJImNsYXNzIjogIkZpbGUiLA0KCQkicGF0aCI6ICIuL3RtcC9FUlIwMzA4ODZfMl9mYXN0cWMuaHRtbCINCgl9LCB7DQoJCSJjbGFzcyI6ICJGaWxlIiwNCgkJInBhdGgiOiAiLi90bXAvRVJSMDMwODg2XzFfZmFzdHFjLmh0bWwiDQoJfV0sDQoJInppcHBlZF9maWxlcyI6IFt7DQoJCSJjbGFzcyI6ICJGaWxlIiwNCgkJInBhdGgiOiAiLi90bXAvRVJSMDMwODg2XzJfZmFzdHFjLnppcCINCgl9LCB7DQoJCSJjbGFzcyI6ICJGaWxlIiwNCgkJInBhdGgiOiAiLi90bXAvRVJSMDMwODg2XzFfZmFzdHFjLnppcCINCgl9XQ0KfQ== --docker-uri quay.io/wshands/fastqc:latest --dockstore-url https://dockstore.org/containers/quay.io/wshands/fastqc --workflow-type sequence_upload_qc_report --parent-uuid aea8dccd-a1b3-50c6-b92f-a8b470743d84 --vm-instance-type m4.4xlarge --vm-region us-west-2 --vm-instance-cores 16 --vm-instance-mem-gb 64 --vm-location aws --tmpdir <path with lots of storage>
+    python DockstoreRunner.py --redwood-path `pwd`/ucsc-storage-client --redwood-token `cat accessToken` --redwood-host storage.ucsc-cgl.org --json-encoded ew0KCSJmYXN0cV9maWxlcyI6IFt7DQoJCSJjbGFzcyI6ICJGaWxlIiwNCgkJInBhdGgiOiAicmVkd29vZDovL3N0b3JhZ2UudWNzYy1jZ2wub3JnL2ZiM2RkODVkLTM2N2UtNWI4ZC05OTI5LTk1MTY0MDg5ZDEwZi83M2YyMzYyNS04ZTU1LTU0MDgtOWY0ZS1hMmRlZDg0MGE2NWQvTkExMjg3OC1OR3YzLUxBQjEzNjAtQV8xLmZhc3RxLmd6Ig0KCX1dLA0KCSJ6aXBwZWRfZmlsZSI6IHsNCgkJImNsYXNzIjogIkZpbGUiLA0KCQkicGF0aCI6ICIvdG1wL2Zhc3RxY19yZXBvcnRzLnRhci5neiINCgl9DQp9 --docker-uri quay.io/briandoconnor/fastqc:0.11.5 --dockstore-url https://dockstore.org/containers/quay.io/briandoconnor/fastqc --workflow-type sequence_upload_qc_report --parent-uuid d2545c4e-dd7d-5a07-b598-e9acba87228f --vm-instance-type m4.4xlarge --vm-region us-west-2 --vm-instance-cores 16 --vm-instance-mem-gb 64 --vm-location aws --tmpdir /datastore
 
-This encoded string corresponds to the contents of `sample.json`.
+This encoded string corresponds to the contents of `sample_fastqc.json`.
 
 To encode and decode online see: https://www.base64encode.org/
 
-## Via Docker
+### Testing Via cwltool
 
-Build the docker image:
+This may be useful for debugging, it's one layer above calling the python script directly.
+
+NOTE: THE ENVIRONMENT VARIABLE TMPDIR MUST BE SET TO A DIRECTORY WITH ENOUGH SPACE TO HOLD INPUT, OUTPUT AND INTERMEDIATE FILES. Otherwise cwltool will use /VAR/SPOOL/CWL by default which may not have enough space.
+
+    cwltool --debug --enable-dev --non-strict --enable-net  <path to>/Dockstore.cwl --redwood-path `pwd`/ucsc-storage-client --redwood-token `cat accessToken` --redwood-host storage.ucsc-cgl.org --json-encoded  ew0KCSJmYXN0cV9maWxlcyI6IFt7DQoJCSJjbGFzcyI6ICJGaWxlIiwNCgkJInBhdGgiOiAicmVkd29vZDovL3N0b3JhZ2UudWNzYy1jZ2wub3JnL2ZiM2RkODVkLTM2N2UtNWI4ZC05OTI5LTk1MTY0MDg5ZDEwZi83M2YyMzYyNS04ZTU1LTU0MDgtOWY0ZS1hMmRlZDg0MGE2NWQvTkExMjg3OC1OR3YzLUxBQjEzNjAtQV8xLmZhc3RxLmd6Ig0KCX1dLA0KCSJ6aXBwZWRfZmlsZSI6IHsNCgkJImNsYXNzIjogIkZpbGUiLA0KCQkicGF0aCI6ICIvdG1wL2Zhc3RxY19yZXBvcnRzLnRhci5neiINCgl9DQp9 --dockstore-uri quay.io/briandoconnor/fastqc:0.11.5 --parent-uuid d2545c4e-dd7d-5a07-b598-e9acba87228f --tmpdir /datastore
+
+### Testing Via Dockstore CLI
+
+Most people, other than developers of this tool, will use the Dockstore CLI to invoke it.  Here's an example running on an Ubutnu box.  Build the docker image:
 
     # patch in /usr/local/lib/python2.7/dist-packages/cwltool
     # make a tmpdir like /datastore
-    docker build -t quay.io/ucsc_cgl/dockstore-tool-runner:1.0.7 .
+    docker build -t quay.io/ucsc_cgl/dockstore-tool-runner:1.0.8 .
     # fill in your JSON from Dockstore.json template as Dockstore.my.json
     mkdir /datastore; chown ubuntu:ubuntu /datastore/
     # local execution
     TMPDIR=/datastore dockstore tool launch --entry Dockstore.cwl --local-entry --json Dockstore.my.json
-    # as root in /datastore 
+    # as root in /datastore
     TMPDIR=/datastore dockstore tool launch --entry ~ubuntu/gitroot/BD2KGenomics/dcc-dockstore-tool-runner/Dockstore.cwl --local-entry --json ~ubuntu/gitroot/BD2KGenomics/dcc-dockstore-tool-runner/Dockstore.my.json
-    # execute published on dockstore
-    dockstore tool launch --entry quay.io/ucsc_cgl/dockstore-tool-runner:1.0.7 --json Dockstore.my.json
+    # execute published on dockstore (this is the way most people will use this tool!)
+    dockstore tool launch --entry quay.io/ucsc_cgl/dockstore-tool-runner:1.0.8 --json Dockstore.my.json
 
-    # running you see it launch
+    # running you see it launch the cwltool command, you mind find this useful while debugging
     cwltool --enable-dev --non-strict --enable-net --outdir /datastore/./datastore/launcher-ff6b55b3-52e8-430c-9a70-1ff295332698/outputs/ --tmpdir-prefix /datastore/./datastore/launcher-ff6b55b3-52e8-430c-9a70-1ff295332698/working/ /home/ubuntu/gitroot/BD2KGenomics/dcc-dockstore-tool-runner/Dockstore.cwl /datastore/./datastore/launcher-ff6b55b3-52e8-430c-9a70-1ff295332698/workflow_params.json
 
-## Via cwltool
-NOTE: THE ENVIRONMENT VARIABLE TMPDIR MUST BE SET TO A DIRECTORY WITH ENOUGH SPACE TO HOLD INPUT, OUTPUT AND INTERMEDIATE FILES. Otherwise cwltool will use /VAR/SPOOL/CWL by default which may not have enough space.
-
-    cwltool --debug --enable-dev --non-strict --enable-net  <path to>/Dockstore.cwl --redwood-path `pwd`/ucsc-storage-client --redwood-token `cat accessToken` --redwood-host storage2.ucsc-cgl.org --json-encoded  ew0KCSJmYXN0cV9maWxlIjogWw0KDQoJCXsNCgkJCSJjbGFzcyI6ICJGaWxlIiwNCgkJCSJwYXRoIjogInJlZHdvb2Q6Ly9zdG9yYWdlMi51Y3NjLWNnbC5vcmcvOGViZGIwM2EtM2M5OS01ZjMyLTgxMWMtOWQ3NGI4ODE1MWVjLzJlYWRjYzY1LTQ0YWYtNTI3Yy1hMWE3LTIyYzNhNTVkNzM2ZS9FUlIwMzA4ODZfMS5mYXN0cS5neiINCgkJfSwgew0KCQkJImNsYXNzIjogIkZpbGUiLA0KCQkJInBhdGgiOiAicmVkd29vZDovL3N0b3JhZ2UyLnVjc2MtY2dsLm9yZy84ZWJkYjAzYS0zYzk5LTVmMzItODExYy05ZDc0Yjg4MTUxZWMvODM0NTIzZjMtN2RkZi01MDg2LWExNzMtMTA4MDYwYWVlZTc3L0VSUjAzMDg4Nl8yLmZhc3RxLmd6Ig0KCQl9DQoJXQ0KfQ== --dockstore-uri quay.io/wshands/fastqc --parent-uuid id --tmpdir <path with lots of storage>
-
-## Via Dockstore
-NOTE: THE ENVIRONMENT VARIABLE TMPDIR MUST BE SET TO A DIRECTORY WITH ENOUGH SPACE TO HOLD INPUT, OUTPUT AND INTERMEDIATE FILES. Otherwise cwltool (called by dockstore) will use /VAR/SPOOL/CWL by default which may not have enough space.
-
-Sample:
-
-    dockstore tool launch --entry quay.io/ucsc-cgl/DockstoreRunner:latest --json Docstore.json
-
 ## Known Issues
 
     AttributeError: 'str' object has no attribute 'append'

diff --git a/docs/mac_mount.png b/docs/mac_mount.png
diff --git a/metadata.json b/metadata.json
@@ -1,67 +1,49 @@
 
 {
    "version" : "1.0.0",
-   "timestamp" : "2016-11-19T06:01:00.549378",
+   "timestamp" : "2017-02-19T22:46:44.013930",
    "parent_uuids" : [
-      "id"
+      "d2545c4e-dd7d-5a07-b598-e9acba87228f"
    ],
-   "workflow_url" : "https://dockstore.org/containers/quay.io/wshands/fastqc",
-   "workflow_name" : "quay.io/wshands/fastqc",
-   "workflow_version" : "latest",
-   "analysis_type" : "qc",
-   "bundle_uuid" : "3e0d390f-40ca-4c2b-b789-5594042d5fd1",
+   "workflow_url" : "https://dockstore.org/containers/quay.io/briandoconnor/fastqc",
+   "workflow_name" : "quay.io/briandoconnor/fastqc",
+   "workflow_version" : "0.11.5",
+   "analysis_type" : "sequence_upload_qc_report",
+   "bundle_uuid" : "8c334a94-5b94-46ac-98ee-6c95e7446d70",
    "workflow_params" : {
 
    },
    "workflow_outputs" : [
    {
-              "file_path": "ERR030886_2_fastqc.zip",
-              "file_type": "zip",
-              "file_checksum": "sha1$f57bc5f399b216d159df478b1cecea0a75c3cd10",
-              "file_size": 494406
-            }
-            ,{
-              "file_path": "ERR030886_1_fastqc.zip",
-              "file_type": "zip",
-              "file_checksum": "sha1$27fa7c14318ecd7852ca8ecdf96ca6b83fcd4932",
-              "file_size": 425451
-            }
-            ,{
-              "file_path": "ERR030886_2_fastqc.html",
-              "file_type": "html",
-              "file_checksum": "sha1$6bb71b3f39c66f5e8664003312b274c53e9e2160",
-              "file_size": 391261
-            }
-            ,{
-              "file_path": "ERR030886_1_fastqc.html",
-              "file_type": "html",
-              "file_checksum": "sha1$d8645a7fe676245481971a7592130f34a6492397",
-              "file_size": 349721
+              "file_path": "fastqc_reports.tar.gz",
+              "file_type": "gz",
+              "file_checksum": "sha1$0f950971ff8352e6f91127c9f8ca01258ce3bffb",
+              "file_size": 644116
             }
 
    ],
    "workflow_inputs" :
-        [{"file_storage_bundle_files": [{"file_type": "fastq.gz", "file_name": "fastq_file", "file_storage_bundle_id": "8ebdb03a-3c99-5f32-811c-9d74b88151ec", "file_path": "ERR030886_1.fastq.gz", "file_storage_id": "2eadcc65-44af-527c-a1a7-22c3a55d736e"}, {"file_type": "fastq.gz", "file_name": "fastq_file", "file_storage_bundle_id": "8ebdb03a-3c99-5f32-811c-9d74b88151ec", "file_path": "ERR030886_2.fastq.gz", "file_storage_id": "834523f3-7ddf-5086-a173-108060aeee77"}], "file_storage_bundle_id": "8ebdb03a-3c99-5f32-811c-9d74b88151ec"}, {"file_storage_bundle_files": [{"file_type": "zip", "file_name": "zipped_files", "file_storage_bundle_id": ".", "file_path": "ERR030886_2_fastqc.zip", "file_storage_id": "tmp"}, {"file_type": "zip", "file_name": "zipped_files", "file_storage_bundle_id": ".", "file_path": "ERR030886_1_fastqc.zip", "file_storage_id": "tmp"}, {"file_type": "html", "file_name": "report_files", "file_storage_bundle_id": ".", "file_path": "ERR030886_2_fastqc.html", "file_storage_id": "tmp"}, {"file_type": "html", "file_name": "report_files", "file_storage_bundle_id": ".", "file_path": "ERR030886_1_fastqc.html", "file_storage_id": "tmp"}], "file_storage_bundle_id": "."}]
+        [{"file_storage_bundle_files": [{"file_type": "fastq.gz", "file_name": "fastq_files", "file_storage_bundle_id": "fb3dd85d-367e-5b8d-9929-95164089d10f", "file_path": "NA12878-NGv3-LAB1360-A_1.fastq.gz", "file_storage_id": "73f23625-8e55-5408-9f4e-a2ded840a65d"}], "file_storage_bundle_id": "fb3dd85d-367e-5b8d-9929-95164089d10f"}]
    ,
 
    "qc_metrics" : {
    },
    "timing_metrics" : {
       "step_timing" : {
          "download" : {
-            "stop_time_utc" : "2016-11-19T06:00:52.477083",
-            "walltime_seconds" : 8,
-            "start_time_utc" : "2016-11-19T06:01:00.549311"
+            "stop_time_utc" : "2017-02-19T22:41:14.289183",
+            "walltime_seconds" : 10,
+            "start_time_utc" : "2017-02-19T22:41:24.935347"
          },
          "tool_run" : {
-            "stop_time_utc" : "2016-11-19T06:01:00.549366",
-            "walltime_seconds" : 0,
-            "start_time_utc" : "2016-11-19T06:01:00.549376"
+            "stop_time_utc" : "2017-02-19T22:41:24.935411",
+            "walltime_seconds" : 319,
+            "start_time_utc" : "2017-02-19T22:46:44.013903"
          }
       },
-      "overall_stop_time_utc" : "2016-11-19T06:01:00.549378",
-      "overall_start_time_utc" : "2016-11-19T06:00:52.477083",
-      "overall_walltime_seconds" : 8
+      "overall_stop_time_utc" : "2017-02-19T22:46:44.013930",
+      "overall_start_time_utc" : "2017-02-19T22:41:14.289183",
+      "overall_walltime_seconds" : 329
    },
    "host_metrics" : {
       "vm_instance_type" : "m4.4xlarge",