pfcon workflow example

Workflow

Abstract

This page presents a sample workflow, demonstrating JSON payload sent to pfcon and asynchronous status query.

Preconditions

The companion services, pfioh and pman are assumed started on some (typically remote) host and assumed to be network accessible.

Start

Start pfcon on localhost using

pfcon --forever --httpResponse

Run a sample plugin/app

Using pfurl, a typical start message to pfcon could be constructed as follows:

pfurl --verb POST --raw --http 10.17.24.163:5005/api/v1/cmd --httpResponseBodyParse --jsonwrapper 'payload' --msg '
        {   "action": "coordinate",
            "threadAction": true,
            "meta-store": {
                        "meta":         "meta-compute",
                        "key":          "jid"
            },

            "meta-data": {
                "remote": {
                        "key":          "%meta-store"
                },
                "localSource": {
                        "path":         "/hostFS/pfconFS/source/Pictures"
                },
                "localTarget": {
                        "path":         "/hostFS/pfconFS/target/Pictures",
                        "createDir":    true                },
                "specialHandling": {
                        "op":           "dsplugin"
                },
                "transport": {
                    "mechanism":    "compress",
                    "compress": {
                        "encoding": "none",
                        "archive":  "zip",
                        "unpack":   true,
                        "cleanup":  true
                    }
                },
                "service":              "pangea"
            },

            "meta-compute":  {
                "cmd":      "$execshell $selfpath/$selfexec --prefix test- --sleepLength 0 /share/incoming /share/outgoing",
                "auid":     "rudolphpienaar",
                "jid":      "simpledsapp-1",
                "threaded": true,
                "container":   {
                        "target": {
                            "image":            "fnndsc/pl-simpledsapp",
                            "cmdParse":         true
                        },
                        "manager": {
                            "image":            "fnndsc/swarm",
                            "app":              "swarm.py",
                            "env":  {
                                "meta-store":   "key",
                                "serviceType":  "docker",
                                "shareDir":     "%shareDir",
                                "serviceName":  "testService"
                            }
                        }
                },
                "service":              "pangea"
            }
        }
'

Query the status of the job

pfurl --verb POST --raw --http ${HOST_IP}:5005/api/v1/cmd --httpResponseBodyParse --jsonwrapper 'payload' --msg '
        {   "action":           "status",
            "threadAction":     false,
            "meta": {
                "remote": {
                        "key":          "testService"
                }
            }
        }' --quiet --jsonpprintindent 4

NB NB NB

After each workflow call, you need to manually remove the service from the swarm manager, as well as remove the shared directories on the remote server. Also remove the local target directory!

On remote

sudo rm -fr /home/tmp/share/key-simpledsapp-1 ; dss testService ; dks

On local

rm -fr /home/rudolph/tmp/Pictures