You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Building a project that has materialized views which reference dictionary objects can result is lost data.
Materialized views are unusual, in that a materialized view artifact form a previous build, is running throughout any subsequent dbt build process (except from the period in which it is being rebuilt). This means that if a materialized view depends on another artifact that is built by another model, and that artifact is dropped (even temporarily) during its model build, then the currently running materialized view can fail to process a batch of rows and hence data can be lost.
Steps to reproduce
Make a dbt project that contains a materialized view that depends on a dictionary, such that the materialized view processes a large continuous data flow.
Build the project to set the materialized view running
Build the project again (possibly multiple times)
Data will be lost and exceptions will appear in the clickhouse logs
Expected behaviour
The materialized view should not drop data as a result of dictionary not being found
dbt and/or ClickHouse server logs
<Error> executeQuery: Code: 36. DB::Exception: Dictionary ... not found
At a minimum dbt-clickhouse should use the DDL CREATE OR REPLACE DICTIONARY and not DROP DICTIONARY
The text was updated successfully, but these errors were encountered:
Describe the bug
Building a project that has materialized views which reference dictionary objects can result is lost data.
Materialized views are unusual, in that a materialized view artifact form a previous build, is running throughout any subsequent
dbt build
process (except from the period in which it is being rebuilt). This means that if a materialized view depends on another artifact that is built by another model, and that artifact is dropped (even temporarily) during its model build, then the currently running materialized view can fail to process a batch of rows and hence data can be lost.Steps to reproduce
Expected behaviour
The materialized view should not drop data as a result of dictionary not being found
dbt and/or ClickHouse server logs
At a minimum
dbt-clickhouse
should use the DDLCREATE OR REPLACE DICTIONARY
and notDROP DICTIONARY
The text was updated successfully, but these errors were encountered: