-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable retry
support for Microbatch models
#10751
Commits on Sep 19, 2024
-
Add
PartialSuccess
status type and use it for microbatch models wit……h mixed results
Configuration menu - View commit details
-
Copy full SHA for 39b5cd5 - Browse repository at this point
Copy the full SHA 39b5cd5View commit details
Commits on Sep 20, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 35963b4 - Browse repository at this point
Copy the full SHA 35963b4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3d606fa - Browse repository at this point
Copy the full SHA 3d606faView commit details
Commits on Sep 23, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 839173a - Browse repository at this point
Copy the full SHA 839173aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3b98e49 - Browse repository at this point
Copy the full SHA 3b98e49View commit details
Commits on Sep 24, 2024
-
Move
BatchResults
andBatchType
to separate arifacts file to avoi……d circular imports In our next commit we're gonna modify `dbt/contracts/graph/nodes.py` to import the `BatchType` as part of our work to implement dbt retry for microbatch model nodes. Unfortunately, the import in `nodes.py` creates a circular dependency because `dbt/artifacts/schemas/results.py` imports from `nodes.py` and `dbt/artifacts/schemas/run/v5/run.py` imports from that `results.py`. Thus the new import creates a circular import. Now this _shouldn't_ be necessary as nothing in artifacts should import from the rest of dbt-core. However, we do. We should fix this, but this is also out of scope for this segement of work.
Configuration menu - View commit details
-
Copy full SHA for a461ae1 - Browse repository at this point
Copy the full SHA a461ae1View commit details -
Add
PartialSuccess
as a retry-able status, and use batches to retry…… microbatch models
Configuration menu - View commit details
-
Copy full SHA for 0e6f0a8 - Browse repository at this point
Copy the full SHA 0e6f0a8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0389475 - Browse repository at this point
Copy the full SHA 0389475View commit details -
Configuration menu - View commit details
-
Copy full SHA for d8bfa51 - Browse repository at this point
Copy the full SHA d8bfa51View commit details -
Configuration menu - View commit details
-
Copy full SHA for a6ab35d - Browse repository at this point
Copy the full SHA a6ab35dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8d7ab70 - Browse repository at this point
Copy the full SHA 8d7ab70View commit details -
Configuration menu - View commit details
-
Copy full SHA for 93deb32 - Browse repository at this point
Copy the full SHA 93deb32View commit details -
Configuration menu - View commit details
-
Copy full SHA for cc0ce6a - Browse repository at this point
Copy the full SHA cc0ce6aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 43cf129 - Browse repository at this point
Copy the full SHA 43cf129View commit details -
Configuration menu - View commit details
-
Copy full SHA for 708c4d4 - Browse repository at this point
Copy the full SHA 708c4d4View commit details
Commits on Sep 25, 2024
-
Configuration menu - View commit details
-
Copy full SHA for fb6b509 - Browse repository at this point
Copy the full SHA fb6b509View commit details -
Configuration menu - View commit details
-
Copy full SHA for 03c46f6 - Browse repository at this point
Copy the full SHA 03c46f6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 287bd55 - Browse repository at this point
Copy the full SHA 287bd55View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0643124 - Browse repository at this point
Copy the full SHA 0643124View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4004347 - Browse repository at this point
Copy the full SHA 4004347View commit details -
Configuration menu - View commit details
-
Copy full SHA for fb113e3 - Browse repository at this point
Copy the full SHA fb113e3View commit details -
When running a microbatch model with supplied batches, force non full…
…-refresh behavior This is necessary because of retry. Say on the initial run the microbatch model succeeds on 97% of it's batches. Then on retry it does the last 3%. If the retry of the microbatch model executes in full refresh mode it _might_ blow away the 97% of work that has been done. This edge case seems to be adapter specific.
Configuration menu - View commit details
-
Copy full SHA for 14e8d53 - Browse repository at this point
Copy the full SHA 14e8d53View commit details -
Only pass batches to retry for microbatch model when there was a Part…
…ialSuccess In the previous commit we made it so that retries of microbatch models wouldn't run in full refresh mode when the microbatch model to retry has batches already specified from the prior run. This is only problematic when the run being retried was a full refresh AND all the batches for a given microbatch model failed. In that case WE DO want to do a full refresh for the given microbatch model. To better outline the problem, consider the following: * a microbatch model had a begin of `2020-01-01` and has been running this way for awhile * the begin config has changed to `2024-01-01` and dbt run --full-refresh gets run * every batch for an microbatch model fails * on dbt retry the the relation is said to exist, and the now out of range data (2020-01-01 through 2023-12-31) is never purged To avoid this, all we have to do is ONLY pass the batch information for partially successful microbatch models. Note: microbatch models only have a partially successful status IFF they have both successful and failed batches.
Configuration menu - View commit details
-
Copy full SHA for a6a1ef8 - Browse repository at this point
Copy the full SHA a6a1ef8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 61108ee - Browse repository at this point
Copy the full SHA 61108eeView commit details -
Configuration menu - View commit details
-
Copy full SHA for ee41265 - Browse repository at this point
Copy the full SHA ee41265View commit details -
Configuration menu - View commit details
-
Copy full SHA for b3552f7 - Browse repository at this point
Copy the full SHA b3552f7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8cc95d4 - Browse repository at this point
Copy the full SHA 8cc95d4View commit details -
Configuration menu - View commit details
-
Copy full SHA for ee338d0 - Browse repository at this point
Copy the full SHA ee338d0View commit details -
Configuration menu - View commit details
-
Copy full SHA for d1e6a94 - Browse repository at this point
Copy the full SHA d1e6a94View commit details
Commits on Sep 26, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 4980627 - Browse repository at this point
Copy the full SHA 4980627View commit details -
Configuration menu - View commit details
-
Copy full SHA for 415f695 - Browse repository at this point
Copy the full SHA 415f695View commit details