Goblet Workflows (Beta)

Goblet Workflows is a wrapper around GCP Workflows, which is a fully-managed orchestration platform that executes services in an order that you define: a workflow. These workflows can combine services including custom services hosted on Cloud Run or Cloud Functions, Google Cloud services such as Cloud Vision AI and BigQuery, and any HTTP-based API.

While GCP Workflows is powerful, it requires learning a complex yaml DSL, which makes it hard to manage and scale, and has a steap learning curve to get started.

Goblet Workflows uses similar syntax that you can expect from airflow and prefect, to make it easy to write, deploy, and manager your workflows, while leveraging the severless nature of GCP workflows.

See below for a simple workflow example.

from goblet_workflows.workflow import Workflow
from goblet_workflows.steps import AssignStep, BQStep

w = Workflow("bigquery-step-example", params=["column"])

upack_args = AssignStep(
    w,
    "upack_args",
    column="${column}"
)

bq_step = BQStep(w, "bq_step", "select {column} from DATASET limit 1")

bq_step_2 = BQStep(w, "bq_step_2", "select colomn2 from DATASET2 where column1 = {bq_step[0]} ")

upack_args > bq_step > bq_step_2

Not to deploy you simply have to run the command goblet_workflows deploy, which will yield the workflow below.

Getting started

To install run:

pip install goblet_workflows

To deploy a workflow create a main.py file where you define your workflow and steps.

Then run

goblet_workflows deploy

To view the yaml run

goblet_workflows print

To execute a workflow run

goblet_workflows execute

To execute a workflow run with an arg

goblet_workflows execute --arg '{"project_id":"PROJECT","var1":"xx"}'

Examples

Goblet Workflow Examples

Features

• Generic Steps
• Branches
• Deployment of Workflows and Schedules
• GCP Connectors
• Execute Workflows

GCP Workflows

Overview

Syntax Cheat Sheet

Issues

Please file any issues, bugs or feature requests as an issue on our GitHub page.

Roadmap

☑ Http Steps
☑ GCP Generic Connectors
☑ BQ Connector
☑ Dataform Connector
☑ Deploy
☑ Deploy Arbitrary File
☑ Execute
☐ Delete
☑ Schedule
☐ Conditions
☐ Subworkflows
☐ Parameter typing

Want to Contribute

If you would like to contribute to the library (e.g. by improving the documentation, solving a bug or adding a cool new feature) submit a pull request.