FIX: Load _history tables from temp table #29
Conversation
| cdc_df = dataframe_from_merged_csv(merge_csv, dfm_object) | ||
|
|
||
| key_columns = [col["name"].lower() for col in self.etl_status.last_schema if col["primaryKeyPos"] > 0] | ||
| # Load records into _history table |
There was a problem hiding this comment.
update comment here.
q: is _load a new type of table at this point? we need to communicate to folks who are using the database what it is, so they can ignore if need be.
There was a problem hiding this comment.
If it is just a table to load stuff, maybe truncating at the end as well? I want it to be obvious that it shouldn't be used by data analysts.
There was a problem hiding this comment.
It's sort of a "temp" table. It gets created by the dmap_import application when it starts to load a QLIK table and then gets dropped at the end. So DB users would only ever see it while the ETL operations is actively running.
| @@ -81,7 +81,7 @@ def get_cdc_gz_csvs(etl_status: TableStatus, table: str) -> List[str]: | |||
| cdc_csvs = s3_list_cdc_gz_objects(S3_ARCHIVE, snapshot_prefix, min_ts=etl_status.last_cdc_ts) | |||
|
|
|||
| # filter error files from table folder | |||
| for csv_file in s3_list_cdc_gz_objects(S3_ERROR, table, min_ts=etl_status.last_cdc_ts): | |||
There was a problem hiding this comment.
Added the fix for the S3 Permission error we were seeing as well
This change modifies the loading of
_historytable.Rather than load a CSV directly into the
_histroytable, we will first load records into the_loadtable and then INSERT into the_historytable with aON CONFLICTquery.This will make
_historytable loads idempotent, in case the same history record is loaded more than once.