Skip to content

Commit

Permalink
Document incremental migration option
Browse files Browse the repository at this point in the history
Signed-off-by: Alice Purcell <alicederyn@gmail.com>
  • Loading branch information
alicederyn committed Oct 16, 2024
1 parent 5969aa5 commit 461785e
Show file tree
Hide file tree
Showing 3 changed files with 123 additions and 0 deletions.
37 changes: 37 additions & 0 deletions docs/user-guides/decorators.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,3 +170,40 @@ We can simply call the script templates, passing the input objects in.

For more complex examples, including use of a dag, see
[the "experimental" examples](../examples/workflows/experimental/new_dag_decorator_params.md).

## Incremental workflow migration

If you have a larger workflow you want to migrate to decorator syntax, you can enable a hybrid mode where Pydantic types can be passed to functions in a Steps/DAG context block, intermixed with calls that pass dictionaries. This will allow you to make smaller changes, and verify that the generated YAML remains the same. For example:

```py
from hera.shared import global_config
from hera.workflows import Input, Output, Steps, Workflow, script

global_config.experimental_features["context_manager_pydantic_io"] = True

class MyInput(Input):
value: int

class MyOutput(Output):
value: int

# Function migrated to Pydantic I/O
@script()
def double(input: MyInput) -> MyOutput:
return MyOutput(value = input.value * 2)

# Not yet migrated to Pydantic I/O
@script()
def print_value(value: int) -> None:
print("Value was", value)

# Not yet migrated to decorator syntax
with Workflow(name="my-template") as w:
with Steps(name="steps"):
# Can now pass Pydantic types to/from functions
first_step = double(Input(value=5))
# Results can be passed into non-migrated functions
print_value(arguments={"value": first_step.value})
```

This feature is turned on by a different experimental flag, as we recommend only using this as a temporary stop-gap during a migration. Once you have fully migrated, you can disable the flag again to verify you are no longer using hybrid mode.
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: my-template
spec:
templates:
- name: steps
steps:
- - name: double
template: double
- - arguments:
parameters:
- name: value
value: '{{steps.double.outputs.parameters.value}}'
name: print-value
template: print-value
- inputs:
parameters:
- name: value
name: double
outputs:
parameters:
- name: value
script:
command:
- python
image: python:3.9
source: |-
import os
import sys
sys.path.append(os.getcwd())
import json
try: value = json.loads(r'''{{inputs.parameters.value}}''')
except: value = r'''{{inputs.parameters.value}}'''
return MyOutput(value=input.value * 2)
- inputs:
parameters:
- name: value
name: print-value
script:
command:
- python
image: python:3.9
source: |-
import os
import sys
sys.path.append(os.getcwd())
import json
try: value = json.loads(r'''{{inputs.parameters.value}}''')
except: value = r'''{{inputs.parameters.value}}'''
print('Value was', value)
33 changes: 33 additions & 0 deletions examples/workflows/experimental/incremental_workflow_migration.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
from hera.shared import global_config
from hera.workflows import Input, Output, Steps, Workflow, script

global_config.experimental_features["context_manager_pydantic_io"] = True


class MyInput(Input):
value: int


class MyOutput(Output):
value: int


# Function migrated to Pydantic I/O
@script()
def double(input: MyInput) -> MyOutput:
return MyOutput(value=input.value * 2)


# Not yet migrated to Pydantic I/O
@script()
def print_value(value: int) -> None:
print("Value was", value)


# Not yet migrated to decorator syntax
with Workflow(name="my-template") as w:
with Steps(name="steps"):
# Can now pass Pydantic types to/from functions
first_step = double(Input(value=5))
# Results can be passed into non-migrated functions
print_value(arguments={"value": first_step.value})

0 comments on commit 461785e

Please sign in to comment.