optimizing large workflow execution

[xzhub]

I am using minikube 0.28 to start a kubenetes 1.64, and using python 2 with reana 0.20.
I could run through the common-workflow-language test with only 3 failures. However, when I were testing with my downloaded CWL file, I couldn't get it run through.

## official test could run through with only 3 failures
$ ./run_test.sh RUNNER=reana-cwl-runner
--- Running conformance test v1.0 on /home/user/p2-reana-cluster/bin/reana-cwl-runner ---
reana-cwl-runner, version 0.2.0
Test [1/132] General test of command line generation
Test [2/132] Test nested prefixes with arrays
Test [3/132] Test nested command line bindings

... ...

Test failed: /home/user/p2-reana-cluster/bin/reana-cwl-runner --outdir=/tmp/tmpuT9_fn --quiet v1.0/fail-unconnected.cwl v1.0/empty.json
Test that parameters that don't appear in the `run` process
inputs are not present in the input object used to run the tool.

Returned zero but it should be non-zero
129 tests passed, 3 failures, 0 unsupported features

1 tool tests failed

For the test script of my own, here is the setup:

$ mkdir /reana
$ cd /reana
## get CWL file
$ wget https://raw.githubusercontent.com/common-workflow-language/workflows/master/tools/bwa-docker.yml
$ wget https://raw.githubusercontent.com/common-workflow-language/workflows/master/tools/bwa-Dockerfile
$ wget https://raw.githubusercontent.com/common-workflow-language/workflows/master/tools/bwa-index.cwl

## configure file
$ cat index.yml
sequences:
    type: File
    path: /reana/test.fa

## data file
$ cat test.fa
>chr1
taaccctaaccctaaccctaaccctaaccctaaccctaaccctaacccta
accctaaccctaaccctaaccctaaccctaaccctaaccctaaccctaac
cctaacccaaccctaaccctaaccctaaccctaaccctaaccctaacccc

## run it
$ reana-cwl-runner bwa-index.cwl index.yml
/home/user/p2-reana-cluster/bin/reana-cwl-runner 1.0.20180721142728
INFO:cwltool:/home/user/p2-reana-cluster/bin/reana-cwl-runner 1.0.20180721142728
Resolved 'bwa-index.cwl' to 'file:///reana/bwa-index.cwl'
INFO:cwltool:Resolved 'bwa-index.cwl' to 'file:///reana/bwa-index.cwl'
Traceback (most recent call last):
  File "/home/user/p2-reana-cluster/lib/python2.7/site-packages/reana_client/cli/cwl_runner.py", line 92, in cwl_runner
    'cwl-runner')
  File "/home/user/p2-reana-cluster/lib/python2.7/site-packages/reana_client/api/client.py", line 173, in create_workflow
    raise e
RequestsFutureAdapterConnectionError
ERROR:root:None

[tiborsimko]

Thanks for trying out CWL on REANA!

Here are three remarks:

(1) I can run your test well locally with cwltool on a vanilla system (i.e. outside of the REANA platform) if I change sequences to use class: File in index.yml.

(2) I have noticed that Docker runtime options are commented out in your sources... Note that this is a necessary pre-requirement for using REANA, because we are running computational workflows via containerisation. The mentioned issue was fixed in upstream cwltool, so the line could be uncommented in bwa-index.cwl.

(3) Finally, please use relative path, beacuse the absolute paths are not guaranteed to exist... Hence the input file should look like:

$ cat index.yml       
sequences:
    class: File
    path: test.fa

After these three changes I can run your example fine on the REANA latest master version.

First I create a reana.yaml file looking like this:

$ cat reana.yaml 
inputs:
  parameters:
    input: index.yml
environments:
  - type: docker
    image: scidap/bwa:v0.7.12
workflow:
  type: cwl
  file: bwa-index.cwl

and then use the usual reana-client commands to run the example.

This produces output files like:

$ reana-client list                                                
NAME                            SIZE   LAST-MODIFIED                   
inputs.json                     51     2018-08-03 17:15:40.737827+00:00
workflow.json                   1477   2018-08-03 17:15:40.737827+00:00
test.fa                         159    2018-08-03 17:15:45.016826+00:00
cwl/docker_outdir/test.fa.sa    128    2018-08-03 17:15:45.013826+00:00
cwl/docker_outdir/test.fa.bwt   244    2018-08-03 17:15:45.000826+00:00
cwl/docker_outdir/test.fa.amb   8      2018-08-03 17:15:45.013826+00:00
cwl/docker_outdir/test.fa.ann   31     2018-08-03 17:15:45.013826+00:00
cwl/docker_outdir/test.fa.pac   39     2018-08-03 17:15:45.013826+00:00
cwl/docker_outdir/test.fa       159    2018-08-03 17:15:35.288830+00:00
cwl/outdir/6r6x_x/test.fa       159    2018-08-03 17:15:45.016826+00:00

P.S. If you are using REANA v0.2.0, your output will look slightly different (e.g. reana-client outputs list`) because of master branch updates; but the example should work pretty much in the similar way.

P.S. Note also that for REANA v0.2.0 an older version of minikube is recommended, e.g. kubectl 1.9.1 and minikube 0.23.0.

[xzhub]

Thanks @tiborsimko, I made some progress. After I fixed the absolute path issue in the index.yml, my command run through. However, There are still a couple issues:

1. My dataset usually is very large 100~500GB, how can I avoid transfer data into and download results out of reana cluster? Can I mount a folder so that reana cluster can read in data file/ write out results file in this folder?
2. Some INFO are mis-labeled as ERROR, for example: "ERROR:root:{u'message': u'File successfully transferred'}. Am I understanding it right? Or it is actually an error?

$ reana-cwl-runner bwa-index.cwl index.yml
/home/user/p2-reana-cluster/bin/reana-cwl-runner 1.0.20180721142728
INFO:cwltool:/home/user/p2-reana-cluster/bin/reana-cwl-runner 1.0.20180721142728
Resolved 'bwa-index.cwl' to 'file:///reana/bwa-index.cwl'
INFO:cwltool:Resolved 'bwa-index.cwl' to 'file:///reana/bwa-index.cwl'
ERROR:root:{u'message': u'Workflow workspace created', u'workflow_id': u'3df25f22-4b60-490c-b8ad-f6d2c42991b1', u'workflow_name': u'cwl-runner.3'}
ERROR:root:{u'message': u'File successfully transferred'}
ERROR:root:Transferred file: /reana/test.fa
ERROR:root:{u'status': u'running', u'workflow_name': u'cwl-runner.3', u'workflow_id': u'3df25f22-4b60-490c-b8ad-f6d2c42991b1', u'user': u'00000000-0000-0000-0000-000000000000', u'organization': u'default', u'message': u'Workflow successfully launched'}
ERROR:root:Polling workflow logs
ERROR:root:/usr/local/bin/celery 0.2.0 with cwltool 1.0.20180326152342Search path is [u'/data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/inputs/workflow.json#main', u'/root/.local/share/commonwl/workflow.json#main', u'/usr/local/share/commonwl/workflow.json#main', u'/usr/share/commonwl/workflow.json#main']Resolved 'workflow.json#main' to 'file:///data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/inputs/workflow.json#main'
ERROR:root:Polling workflow logs
ERROR:root:[job workflow.json] initializing from file:///data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/inputs/workflow.json[job workflow.json] {
    "sequences": {
        "class": "File",
        "location": "file:///data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/inputs/test.fa",
        "size": 159,
        "basename": "test.fa",
        "nameroot": "test",
        "nameext": ".fa"
    }
}[job workflow.json] path mappings is {
    "file:///data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/inputs/test.fa": [
        "/data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/inputs/test.fa",
        "/data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/workspace/cwl/docker_outdir/test.fa",
        "File",
        false
    ]
}[job workflow.json] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "bwa"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "index"
    },
    {
        "prefix": "-a",
        "position": [
            0,
            "algorithm"
        ],
        "datum": null
    },
    {
        "prefix": "-b",
        "position": [
            0,
            "block_size"
        ],
        "datum": null
    },
    {
        "position": [
            4,
            "sequences"
        ],
        "valueFrom": "$(self.basename)",
        "datum": {
            "class": "File",
            "location": "file:///data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/inputs/test.fa",
            "size": 159,
            "basename": "test.fa",
            "nameroot": "test",
            "nameext": ".fa",
            "path": "/data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/workspace/cwl/docker_outdir/test.fa",
            "dirname": "/data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/workspace/cwl/docker_outdir"
        }
    }
][job workflow.json] initial work dir {
    "file:///data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/inputs/test.fa": [
        "/data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/inputs/test.fa",
        "/data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/workspace/cwl/docker_outdir/test.fa",
        "File",
        true
    ]
}
ERROR:root:Polling workflow logs
ERROR:root:Polling workflow logs
ERROR:root:Polling workflow logs
ERROR:root:Copying /data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/inputs/test.fa to /data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/outputs/test.faFinal process status is success{
    "output": {
        "checksum": "sha1$10420f31c503a377d1750a2b64fea62e35fa505b",
        "basename": "test.fa",
        "location": "file:///data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/outputs/test.fa",
        "secondaryFiles": [],
        "path": "/data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/outputs/test.fa",
        "class": "File",
        "size": 159
    }
}

{
    "output": {
        "checksum": "sha1$10420f31c503a377d1750a2b64fea62e35fa505b",
        "basename": "test.fa",
        "location": "file:///data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/outputs/test.fa",
        "secondaryFiles": [],
        "path": "/data/00000000-0000-0000-0000-000000000000/analyses/3df25f22-4b60-490c-b8ad-f6d2c42991b1/outputs/test.fa",
        "class": "File",
        "size": 159
    }
}