-
Notifications
You must be signed in to change notification settings - Fork 47
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Prefect 3 #2221
Comments
The entire test suite runs to completion if the following small change is made: changing Lines 44 to 56 in 56a923a
In other words, it works perfectly fine if we don't The biggest problem actually seems to be that you lose the DAG if some tasks are submitted and others aren't. |
This might be related (and if so, it might be worth trying the tests again with the latest prefect3): PrefectHQ/prefect#14036 |
Note to self: this does not seem related to PrefectHQ/prefect#14036 since the errors persist in 3.0.0rc15. |
We know that the following always works: from prefect import task, flow
@task
def my_task():
return 1
@flow
def my_subflow(dummy):
return 1
@flow
def my_flow():
result = my_task.submit()
return my_subflow(result)
assert my_flow() == 1 But for some reason in Prefect 3, the following doesn't: from prefect.futures import PrefectFuture
from prefect import flow, task
def _patched_getitem(self, index):
@task
def get_item(obj, key):
return obj[key]
return get_item.submit(self, index)
PrefectFuture.__getitem__ = _patched_getitem
@task
def my_task():
return 1
@flow
def my_subflow(dummy):
return 1
@flow
def my_flow():
result = my_task.submit()
return my_subflow(result)
assert my_flow() == 1 Linking to PrefectHQ/prefect#14881. |
Prefect 3 support will be fixed with the upstream Prefect PR I opened: PrefectHQ/prefect#14900. |
Details about the quacc environment
What is the issue?
When using Prefect 3.x, most dynamic workflows in quacc become broken.
How can we easily reproduce the issue?
Alternatively, you can run
pytest tests/prefect/test_syntax.py
to find more examples.In running the above minimal example, you'll find that it stalls completely (if run via
pytest
) or simply returns an infinite loop of errors (if run in an IPython console or REPL).Any Ideas Why?
The root cause is the following monkeypatch:
quacc/src/quacc/__init__.py
Lines 47 to 60 in 797602a
If we comment out
PrefectFuture.__getitem__ = _patched_getitem
, the minimal example will run to completion, albeit at the expense of other tests failing since we have removed the implicit deferral of__getitem__
calls.I believe the problem is likely related to the fact that there are actually several new Perfect future classes like a
PrefectConcurrentFuture
among others. However, I tried playing around with them and wasn't able to make much progress from a quick effort.Tagging @zulissimeta as a head's up (and in case you're feeling adventurous...).
The text was updated successfully, but these errors were encountered: