Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replacing dictionary is not atomic #392

Open
the4thamigo-uk opened this issue Nov 26, 2024 · 0 comments
Open

Replacing dictionary is not atomic #392

the4thamigo-uk opened this issue Nov 26, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@the4thamigo-uk
Copy link
Contributor

the4thamigo-uk commented Nov 26, 2024

Describe the bug

Building a project that has materialized views which reference dictionary objects can result is lost data.

Materialized views are unusual, in that a materialized view artifact form a previous build, is running throughout any subsequent dbt build process (except from the period in which it is being rebuilt). This means that if a materialized view depends on another artifact that is built by another model, and that artifact is dropped (even temporarily) during its model build, then the currently running materialized view can fail to process a batch of rows and hence data can be lost.

Steps to reproduce

  1. Make a dbt project that contains a materialized view that depends on a dictionary, such that the materialized view processes a large continuous data flow.
  2. Build the project to set the materialized view running
  3. Build the project again (possibly multiple times)
  4. Data will be lost and exceptions will appear in the clickhouse logs

Expected behaviour

The materialized view should not drop data as a result of dictionary not being found

dbt and/or ClickHouse server logs

<Error> executeQuery: Code: 36. DB::Exception: Dictionary  ... not found

At a minimum dbt-clickhouse should use the DDL CREATE OR REPLACE DICTIONARY and not DROP DICTIONARY

@the4thamigo-uk the4thamigo-uk added the bug Something isn't working label Nov 26, 2024
the4thamigo-uk added a commit to the4thamigo-uk/dbt-clickhouse that referenced this issue Nov 26, 2024
the4thamigo-uk added a commit to the4thamigo-uk/dbt-clickhouse that referenced this issue Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant