Enable retry support for Microbatch models
#10751
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
QMalcolm
added
Skip Changelog
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #10751 +/- ##
==========================================
+ Coverage 89.00% 89.03% +0.02%
==========================================
Files 181 182 +1
Lines 23126 23195 +69
==========================================
+ Hits 20583 20651 +68
- Misses 2543 2544 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
|
…d circular imports In our next commit we're gonna modify `dbt/contracts/graph/nodes.py` to import the `BatchType` as part of our work to implement dbt retry for microbatch model nodes. Unfortunately, the import in `nodes.py` creates a circular dependency because `dbt/artifacts/schemas/results.py` imports from `nodes.py` and `dbt/artifacts/schemas/run/v5/run.py` imports from that `results.py`. Thus the new import creates a circular import. Now this _shouldn't_ be necessary as nothing in artifacts should import from the rest of dbt-core. However, we do. We should fix this, but this is also out of scope for this segement of work.
… microbatch models
QMalcolm
force-pushed
the
qmalcolm--microbatch-retry
branch
from
c0e16de
to
8d7ab70
Compare
MichelleArk
reviewed
Sep 24, 2024
MichelleArk
reviewed
Sep 24, 2024
MichelleArk
reviewed
Sep 24, 2024
|
I have 2-ish tests to add, and then we should be good to go! |
…-refresh behavior This is necessary because of retry. Say on the initial run the microbatch model succeeds on 97% of it's batches. Then on retry it does the last 3%. If the retry of the microbatch model executes in full refresh mode it _might_ blow away the 97% of work that has been done. This edge case seems to be adapter specific.
…ialSuccess In the previous commit we made it so that retries of microbatch models wouldn't run in full refresh mode when the microbatch model to retry has batches already specified from the prior run. This is only problematic when the run being retried was a full refresh AND all the batches for a given microbatch model failed. In that case WE DO want to do a full refresh for the given microbatch model. To better outline the problem, consider the following: * a microbatch model had a begin of `2020-01-01` and has been running this way for awhile * the begin config has changed to `2024-01-01` and dbt run --full-refresh gets run * every batch for an microbatch model fails * on dbt retry the the relation is said to exist, and the now out of range data (2020-01-01 through 2023-12-31) is never purged To avoid this, all we have to do is ONLY pass the batch information for partially successful microbatch models. Note: microbatch models only have a partially successful status IFF they have both successful and failed batches.
QMalcolm
changed the title
retry support for Microbatch models
|
I still need to open a |
MichelleArk
reviewed
Sep 25, 2024
MichelleArk
reviewed
Sep 25, 2024
core/dbt/events/types.py
Outdated
| @@ -1293,9 +1293,12 @@ def code(self) -> str: | |||
| return "Q012" | |||
|
|
|||
| def message(self) -> str: | |||
| if self.status == "error": | |||
| if self.status == "error": # or 'PARTIAL SUCCESS' in self.status: | |||
There was a problem hiding this comment.
is this comment still relevant?
There was a problem hiding this comment.
It is not, I'll get it fixed
MichelleArk
approved these changes
Sep 25, 2024
QMalcolm
commented
Sep 26, 2024
Comment on lines
+268
to
+270
| "begin": { | ||
| "default": null | ||
| }, |
There was a problem hiding this comment.
It looks like this change was already made in https://github.com/dbt-labs/schemas.getdbt.com, so I'm unsure why this PR is doing it for dbt-core but 🤷
|
The updates to schemas.getdbt.com can be found here dbt-labs/schemas.getdbt.com#61 |
This was referenced Sep 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #10624
Resolves #10715
Problem
dbt was taking an all or nothing approach to microbatch models when running
dbt retry. We want the command to instead be smarter, only retrying batches that failed for a given microbatch modelSolution
PartialSuccessbatch_resultsto the RunResult objectfailedbatches to the corresponding model during theretrytaskPartialSuccessmicrobatch modelChecklist