When retrying microbatch models, propagate prior successful state #10802
+70
−12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #10802 +/- ##
==========================================
+ Coverage 89.17% 89.22% +0.04%
==========================================
Files 183 183
Lines 23382 23388 +6
==========================================
+ Hits 20850 20867 +17
+ Misses 2532 2521 -11
Flags with carried forward coverage won't be shown. Click here to find out more.
|
QMalcolm
force-pushed
the
qmalcolm--fix-microbatch-multiple-retries-edge-case
branch
from
9502273
to
0100a58
Compare
ChenyuLInx
approved these changes
Sep 30, 2024
| @@ -454,7 +454,7 @@ def resource_class(cls) -> Type[HookNodeResource]: | |||
|
|
|||
| @dataclass | |||
| class ModelNode(ModelResource, CompiledNode): | |||
| batches: Optional[List[BatchType]] = None | |||
| batch_info: Optional[BatchResults] = None | |||
There was a problem hiding this comment.
Will this cause a loading error for a previous manifest?
There was a problem hiding this comment.
This has no effect on the written manifest, which is intentional. That is the case because these do not exist on the artifact class for the object
| @@ -481,8 +489,8 @@ def _execute_microbatch_materialization( | |||
| start = microbatch_builder.build_start_time(end) | |||
| batches = microbatch_builder.build_batches(start, end) | |||
| else: | |||
There was a problem hiding this comment.
Can we add a comment here mentioning this only happens during retry?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #10800
Problem
If you invoked
dbt retryon a microbatch model twice, where on the firstdbt retryall the retried batches failed, then on the seconddbt retryinvocation the microbatch model would rerun all the batches that were run on the initialdbt runbeing retried.Solution
When going through multiple
dbt retryinvocations, ensure that the priorsuccessfulbatch information continues to be passed.Checklist