Cannot apply the Prefect deployment boilerplate to a versioned dataset #1427
Replies: 1 comment 9 replies
-
|
Potentially related to this issue |
Beta Was this translation helpful? Give feedback.
-
|
Just gave the suggested code change a try and I get an error regarding the session ID:
Any suggestions for how to go about this? |
Beta Was this translation helpful? Give feedback.
-
|
Ok, the above was fixed by switching to ShelveStore instead of SQLiteStore. The pipeline now crashes at a different section but I'm not sure if this is a mistake in another part of the code. Will try first to fix it myself and update here. Additionally: there is strange behavior now where every single node is treated as a separate pipeline by Prefect. After every task this gets printed:
Is it inherent to the fix where the session is re-used for each task that solved the original problem for this discussion? |
Beta Was this translation helpful? Give feedback.
-
|
Coming back with an update: I cannot get the pipeline working (using the spaceflights tutorial code). It was giving me a
To double-check I've just cloned the suggested solution project and that runs fine, albeit with each node treated as its own pipeline as I mentioned previously. The only difference that I can see now from the shared solution is that my project is using different namespaces for the pipelines. This is exactly as described in the tutorial page for namespacing I should mention that just doing a standard |
Beta Was this translation helpful? Give feedback.
-
|
@alexfurnica, does the sample project work as is in your machine? Also which version of kedro are you using? If you're using 0.18.0, I'm not confident that it would work with all the changes that were made to the framework. This solution in the repo requires all the intermediate datasets to be persisted to disc, so cross check if all the dataset entries are present in your catalog. I tried running the example again and this is the run log that was generated for one Prefect run. |
Beta Was this translation helpful? Give feedback.
-
|
@avan-sh thanks for getting back to me! As I've mentioned in my last post, the sample project works, albeit with a bug that treats every node as a separate run, causing multiple redundant prints: Because my deadlines are a little tight I had to drop Kedro and focus on getting Prefect + great expectations to work. That being said I'll try to adapt this sample project with modular, namespaced pipelines to see if that's the issue. Will write back here if there I've got an update on that 👍 |
Beta Was this translation helpful? Give feedback.
-
Hi there!
I've been going through tutorials for Kedro, Prefect and Great Expectations to try and make a proof-of-concept of an end-to-end ML pipeline that includes data quality monitoring. At this stage, most of the spaceflights tutorial has worked (except the experiment tracking bug on Windows).
I'm currently attempting to run the pipeline as a Prefect flow using the boilerplate script provided in the Kedro docs. The pipeline registers just fine and it runs successfully the first time. The issue is that it fails during subsequent runs if I don't manually restart the agent. The reason it fails is that the
saveattribute of theVersionobject does not get updated in subsequent runs of the same pipeline so it throws aDataSetError:The initial run was at 13:17 UTC, but this run was at 13:28 UTC as seen here (+2 hrs due to time difference):
Is this a known issue? Is there something I'm doing wrong? I've tried digging through the code to better understand but it is above my level of experience. Best I could find is that the
tracking.JSONDataSetis not reinitialized with the correct date, or the cache is never cleared within thesave()method.Would really appreciate any suggestions!
Beta Was this translation helpful? Give feedback.
All reactions