Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable retry support for Microbatch models #10751

Merged
merged 31 commits into from
Sep 26, 2024
Merged

Commits on Sep 19, 2024

  1. Configuration menu
    Copy the full SHA
    39b5cd5 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Configuration menu
    Copy the full SHA
    35963b4 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3d606fa View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Configuration menu
    Copy the full SHA
    839173a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3b98e49 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. Move BatchResults and BatchType to separate arifacts file to avoi…

    …d circular imports
    
    In our next commit we're gonna modify `dbt/contracts/graph/nodes.py` to import the
    `BatchType` as part of our work to implement dbt retry for microbatch model nodes.
    Unfortunately, the import in `nodes.py` creates a circular dependency because
    `dbt/artifacts/schemas/results.py` imports from `nodes.py` and `dbt/artifacts/schemas/run/v5/run.py`
    imports from that `results.py`. Thus the new import creates a circular import. Now this
    _shouldn't_ be necessary as nothing in artifacts should import from the rest of dbt-core.
    However, we do. We should fix this, but this is also out of scope for this segement of work.
    QMalcolm committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    a461ae1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0e6f0a8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0389475 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    d8bfa51 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    a6ab35d View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    8d7ab70 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    93deb32 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    cc0ce6a View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    43cf129 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    708c4d4 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Configuration menu
    Copy the full SHA
    fb6b509 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    03c46f6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    287bd55 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0643124 View commit details
    Browse the repository at this point in the history
  5. Cleanup TODO comment

    QMalcolm committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    4004347 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    fb113e3 View commit details
    Browse the repository at this point in the history
  7. When running a microbatch model with supplied batches, force non full…

    …-refresh behavior
    
    This is necessary because of retry. Say on the initial run the microbatch model
    succeeds on 97% of it's batches. Then on retry it does the last 3%. If the retry
    of the microbatch model executes in full refresh mode it _might_ blow away the
    97% of work that has been done. This edge case seems to be adapter specific.
    QMalcolm committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    14e8d53 View commit details
    Browse the repository at this point in the history
  8. Only pass batches to retry for microbatch model when there was a Part…

    …ialSuccess
    
    In the previous commit we made it so that retries of microbatch models wouldn't
    run in full refresh mode when the microbatch model to retry has batches already
    specified from the prior run. This is only problematic when the run being retried
    was a full refresh AND all the batches for a given microbatch model failed. In
    that case WE DO want to do a full refresh for the given microbatch model. To better
    outline the problem, consider the following:
    
    * a microbatch model had a begin of `2020-01-01` and has been running this way for awhile
    * the begin config has changed to `2024-01-01` and  dbt run --full-refresh gets run
    * every batch for an microbatch model fails
    * on dbt retry the the relation is said to exist, and the now out of range data (2020-01-01 through 2023-12-31) is never purged
    
    To avoid this, all we have to do is ONLY pass the batch information for partially successful microbatch
    models. Note: microbatch models only have a partially successful status IFF they have both
    successful and failed batches.
    QMalcolm committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    a6a1ef8 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    61108ee View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    ee41265 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    b3552f7 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    8cc95d4 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    ee338d0 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    d1e6a94 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. Configuration menu
    Copy the full SHA
    4980627 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    415f695 View commit details
    Browse the repository at this point in the history