-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CT-10785] Microbatch models should respect full_refresh
model config
#10788
[CT-10785] Microbatch models should respect full_refresh
model config
#10788
Conversation
In dbt-core, it is generally expected that values passed via CLI flags take precedence over model level configs. However, `full_refresh` on a model is an exception to this rule, where in the model config takes precedence. This config exists specifically to _prevent_ accidental full refreshes of large incremental models, as doing so can be costly. **_It is actually best practice_** to set `full_refresh=False` on incremental models. Prior to this commit, for microbatch models, the above was not happening. The CLI flag `--full-refresh` was taking precedence over the model config `full_refresh`. That meant that if `--full-refresh` was supplied, then the microbatch model **_would full refresh_** even if `full_refresh=False` was set on the model. This commit solves that problem.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #10788 +/- ##
==========================================
+ Coverage 86.34% 89.09% +2.75%
==========================================
Files 182 183 +1
Lines 23288 23321 +33
==========================================
+ Hits 20107 20779 +672
+ Misses 3181 2542 -639
Flags with carried forward coverage won't be shown. Click here to find out more.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code looks good but I have some questions about the new behavior.
Is this a behavior change for the existing incremental model also?
My understanding is with the new overwrite order, the user can not do a full refresh without changing the code. Is this correct? If yes is this what we want? @graciegoheen
Feel free to dismiss my review if my understanding is wrong.
My understanding was wrong
Good question! I was also confused when I first learned about it 😅 For clarification, this is not new behavior for incremental models. This is already how incremental models operate on the |
Dismissing review as reviewer suggested
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚢
Resolves #10785
Problem
In dbt-core, it is generally expected that values passed via CLI flags take precedence over model level configs. However,
full_refresh
on a model is an exception to this rule, where in the model config takes precedence. This config exists specifically to prevent accidental full refreshes of large incremental models, as doing so can be costly. It is actually best practice to setfull_refresh=False
on incremental models.Prior to this PR, for microbatch models, the above was not happening. The CLI flag
--full-refresh
was taking precedence over the model configfull_refresh
. That meant that if--full-refresh
was supplied, then the microbatch model would full refresh even iffull_refresh=False
was set on the model. This PR solves that problem.Solution
Update
_is_incremental
to prefer themodel.config.full_refresh
value over theRuntimeConfig.args.FULL_REFRESH
value provided by CLI flags.Checklist