bugfix/hubspot-duplicates #5
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -45,14 +45,15 @@ engagement_emails as ( | |||
| engagement_email.email_to_email, | |||
| engagement_email.email_cc_email, | |||
| engagement_email.email_from_email as commenter_email, | |||
| contacts.contact_name as commenter_name | |||
| {{ fivetran_utils.string_agg(field_to_agg="contacts.contact_name", delimiter="','") }} as commenter_name | |||
There was a problem hiding this comment.
I found a few cases where there are multiple contacts associated with an engagement email. This resulted in fannout. The stringagg will ensure all parties are included in the resulting data, but also not cause any fannout.
Comment on lines
+87
to
+99
| engagement_details as ( | ||
| select | ||
| deal_id, | ||
| deal_name, | ||
| url_reference, | ||
| created_on, | ||
| source_relation, | ||
| {{ fivetran_utils.string_agg(field_to_agg="distinct engagement_type", delimiter="', '") }} as engagement_type, | ||
| {{ fivetran_utils.string_agg(field_to_agg="distinct contact_name", delimiter="', '") }} as contact_name, | ||
| {{ fivetran_utils.string_agg(field_to_agg="distinct created_by", delimiter="', '") }} as created_by, | ||
| {{ fivetran_utils.string_agg(field_to_agg="distinct company_name", delimiter="', '") }} as company_name | ||
| from engagement_detail_prep | ||
| group by 1,2,3,4,5 |
There was a problem hiding this comment.
Similar to the previous model. The joins in the prev cte are not 1:1. So we need to do some creative aggregating to make sure we retain all the necessary information, but to not cause any fannouts.
| @@ -1,8 +1,8 @@ | |||
| {{ | |||
| config( | |||
| materialized='table' if unified_rag.is_databricks_sql_warehouse() else 'incremental', | |||
| partition_by = {'field': 'most_recent_chunk_update', 'data_type': 'date', 'granularity': 'month'} | |||
| if target.type not in ['spark', 'databricks'] else ['most_recent_chunk_update'], | |||
| partition_by = {'field': 'update_date', 'data_type': 'date'} | |||
There was a problem hiding this comment.
Ran into some issues with the incremental logic on BQ. These changes helped address those issues although it did require adding a new field (which has been documented and docs regen'd).
fivetran-avinash
approved these changes
Oct 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR to address some duplicate HubSpot records uncovered when testing the model on a new schema. Additionally, added unique and not null tests to the final unified rag output model.