Scio Dataflow flex template example

Usage

Setup

set gcpProject := "<YOUR PROJECT>"
set gcpRegion := "europe-west1"

Creating the template

sbt createFlexTemplate

Will build the docker image and publish it to Google Container Registry

JSON template will be uploaded to gs://<YOUR PROJECT>/dataflow/templates/flex-template.json

Example output:

{
    "image": "gcr.io/<YOUR PROJECT>/dataflow/templates/flex-template:0.1.0-SNAPSHOT",
    "metadata": {
        "name": "WordCount Example",
        "parameters": [
            {
                "helpText": "GCS input text file",
                "label": "GCS input text file",
                "name": "input"
            },
            {
                "helpText": "GCS output text file",
                "label": "GCS output text file",
                "name": "output"
            }
        ]
    },
    "sdkInfo": {
        "language": "JAVA"
    }
}

Triggering a run!

sbt runFlextTemplate input=gs://dataflow-samples/shakespeare/kinglear.txt output=gs://<OUTPUT>

⚠️ This is a somewhat opinionated example, check build.sbt and modify it to your own needs.