Add suffix to duplicate passthrough fields (#15)

* add under the hood updates * add conversion metrics to end models * docs and add passthrough * update passthrough * changelog and versioining * readme and docs * add decisionlog about discrepancies across different grains * changelog * realized we can remove total_items from the transforms * add validation tests * fix * new schema * update config * change schema for all * rm horizontal conversions test * add decision log entry for different grains * send to bk * Validation tests passing * tests * polishing * buildkite * seed data type * fix run steps * changelog * reword * joe feedback * source package is up on hub * getting there * avinash feedback --------- Co-authored-by: Renee Li <renee.li@fivetran.com>
fivetran · Nov 13, 2024 · 2363c9c · 2363c9c
1 parent 52bdeb6
commit 2363c9c
Show file tree

Hide file tree

Showing 19 changed files with 140 additions and 102 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,17 @@
+# dbt_reddit_ads v0.3.1
+[PR #15](https://github.com/fivetran/dbt_reddit_ads/pull/15) includes the following updates:
+
+## Bug Fix
+- Ensures the addition of conversion metrics in [v0.3.0](https://github.com/fivetran/dbt_reddit_ads/blob/main/CHANGELOG.md#dbt_reddit_ads-v030) is truly backwards compatible and will avoid duplicate column errors regardless of any pre-existing configurations.
+  - Specifically, if you were previously utilizing [passthrough column](https://github.com/fivetran/dbt_reddit_ads?tab=readme-ov-file#passing-through-additional-metrics) variables to include fields called `conversions`, `view_through_conversions`, `total_value`, or `total_items`, the package's version of these fields will take precedence. Your fields will be included as well, but suffixed with a `_c`.
+
+## Under the Hood
+- Creates the `reddit_ads_persist_pass_through_columns` macro to support the above behavior. 
+- Updates consistency validation tests (maintainers only) to include conversion metric comparisons.
+
+## Documentation
+- Documented `_c` bug fix solution in the [DECISIONLOG](https://github.com/fivetran/dbt_reddit_ads/blob/main/DECISIONLOG.md). 
+
 # dbt_reddit_ads v0.3.0
 [PR #13](https://github.com/fivetran/dbt_reddit_ads/pull/13) includes the following **BREAKING CHANGE** updates:
 

diff --git a/DECISIONLOG.md b/DECISIONLOG.md
@@ -1,4 +1,13 @@
 ## Why don't metrics add up across different grains (Ex. ad level vs campaign level)?
 When aggregating metrics like clicks and spend across different grains, discrepancies can arise due to differences in how data is captured, grouped, or attributed at each grain. For example, certain actions or costs might be attributed differently at the ad, campaign, or ad group level, leading to inconsistencies when rolled up. Additionally, for example, at the keyword grain, where a keyword can belong to multiple ad groups, aggregations can lead to over counting. Conversely, some ads may only be represented at the ad group level, rather than individual ad levels, leading to under counting at the ad grain.
 
-This is a reason why we have broken out the ad reporting packages into separate hierarchical end models (Ad, Ad Group, Campaign, and more). Because if we only used ad-level reports, we could be missing data.
+This is a reason why we have broken out the ad reporting packages into separate hierarchical end models (Ad, Ad Group, Campaign, and more). Because if we only used ad-level reports, we could be missing data.
+
+## Conversion Passthrough Columns
+In [v0.3.0](https://github.com/fivetran/dbt_reddit_ads/releases/tag/v0.3.0) of the package, we introduced the following conversion metrics, all coming from Reddit Ads source `<entity>_conversions_report` tables:
+- `conversions` (aliased from `click_through_conversion_attribution_window_month`): Total attributed click-through conversions for the given month-long window.
+- `view_through_conversions` (aliased from `view_through_conversion_attribution_window_month`): Total attributed view-through conversions for the given month-long window.
+- `total_value`: Total monetary value associated with a conversion event.
+- `total_items`: Total number of items involved in a conversion event.
+
+In [v0.3.1](https://github.com/fivetran/dbt_reddit_ads/releases/tag/v0.3.1), we ensured that the addition of these fields was truly backwards compatible and would avoid duplicate column errors regardless of whatever configurations you previously had in place. Specifically, if you were previously utilizing [passthrough column](https://github.com/fivetran/dbt_reddit_ads?tab=readme-ov-file#passing-through-additional-metrics) variables to include fields called `conversions`, `view_through_conversions`, `total_value`, or `total_items`, the package's version of these fields will take precedence. Your fields will be included as well, but suffixed with a `_c`. 
diff --git a/README.md b/README.md
@@ -126,6 +126,16 @@ vars:
       - name: "new_custom_field"
         alias: "custom_field"
       - name: "a_second_field"
+    reddit_ads__account_conversions_passthrough_metrics:
+      - name: "view_through_conversion_attribution_window_week"
+        alias: view_through_conversion_week
+    reddit_ads__ad_group_conversions_passthrough_metrics:
+      - name: "view_through_conversion_attribution_window_week"
+    reddit_ads__ad_conversions_passthrough_metrics:
+      - name: "view_through_conversion_attribution_window_week"
+        alias: view_through_conversion_week
+    reddit_ads__campaign_conversions_passthrough_metrics:
+      - name: "view_through_conversion_attribution_window_week"
 ```
 #### Change the build schema
 By default, this package builds the Reddit Ads staging models (12 views, 12 tables) within a schema titled (`<target_schema>` + `_reddit_ads_source`) and your Reddit Ads modeling models (5 tables) within a schema titled (`<target_schema>` + `_reddit_ads`) in your destination. If this is not where you would like your Reddit Ads data to be written to, add the following configuration to your root `dbt_project.yml` file:

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'reddit_ads'
-version: '0.3.0'
+version: '0.3.1'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 vars:

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/index.html b/docs/index.html
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'reddit_ads_integration_tests'
-version: '0.3.0'
+version: '0.3.1'
 profile: 'integration_tests'
 config-version: 2
 
@@ -20,6 +20,25 @@ vars:
 
   reddit_ads__ad_conversions_passthrough_metrics: 
     - name: 'view_through_conversion_attribution_window_week'
+  reddit_ads__account_passthrough_metrics: 
+    - name: conversions
+    - name: app_install_metrics_install
+      alias: installs
+    - name: view_through_conversions
+  reddit_ads__campaign_passthrough_metrics: 
+    - name: conversions
+    - name: app_install_metrics_install
+      alias: installs
+    - name: total_value
+  reddit_ads__ad_group_passthrough_metrics:
+    - name: conversions
+    - name: app_install_metrics_install
+      alias: installs
+    - name: total_items
+  reddit_ads__ad_passthrough_metrics:
+    - name: conversions
+    - name: app_install_metrics_install
+      alias: installs
 
   # For validation testing
   reddit_ads__conversion_event_types:

diff --git a/integration_tests/tests/consistency/consistency_account_report.sql b/integration_tests/tests/consistency/consistency_account_report.sql
@@ -8,11 +8,11 @@ with prod as (
         account_id,
         sum(clicks) as clicks, 
         sum(impressions) as impressions,
-        sum(spend) as spend
-        {# sum(conversions) as conversions,
+        sum(spend) as spend,
+        sum(conversions) as conversions,
         sum(view_through_conversions) as view_through_conversions,
         sum(total_value) as total_value,
-        sum(total_items) as total_items #}
+        sum(total_items) as total_items
     from {{ target.schema }}_reddit_ads_prod.reddit_ads__account_report
     group by 1
 ),
@@ -22,11 +22,11 @@ dev as (
         account_id,
         sum(clicks) as clicks, 
         sum(impressions) as impressions,
-        sum(spend) as spend
-        {# sum(conversions) as conversions,
+        sum(spend) as spend,
+        sum(conversions) as conversions,
         sum(view_through_conversions) as view_through_conversions,
         sum(total_value) as total_value,
-        sum(total_items) as total_items #}
+        sum(total_items) as total_items
     from {{ target.schema }}_reddit_ads_dev.reddit_ads__account_report
     group by 1
 ),
@@ -39,15 +39,15 @@ final as (
         prod.impressions as prod_impressions,
         dev.impressions as dev_impressions,
         prod.spend as prod_spend,
-        dev.spend as dev_spend
-        {# prod.conversions as prod_conversions,
+        dev.spend as dev_spend,
+        prod.conversions as prod_conversions,
         dev.conversions as dev_conversions,
         prod.view_through_conversions as prod_view_through_conversions,
         dev.view_through_conversions as dev_view_through_conversions,
         prod.total_value as prod_total_value,
         dev.total_value as dev_total_value,
         prod.total_items as prod_total_items,
-        dev.total_items as dev_total_items #}
+        dev.total_items as dev_total_items
     from prod
     full outer join dev 
         on dev.account_id = prod.account_id
@@ -59,7 +59,7 @@ where
     abs(prod_clicks - dev_clicks) >= .01
     or abs(prod_impressions - dev_impressions) >= .01
     or abs(prod_spend - dev_spend) >= .01
-    {# or abs(prod_conversions - dev_conversions) >= .01
+    or abs(prod_conversions - dev_conversions) >= .01
     or abs(prod_view_through_conversions - dev_view_through_conversions) >= .01
     or abs(prod_total_value - dev_total_value) >= .01
-    or abs(prod_total_items - dev_total_items) >= .01 #}
+    or abs(prod_total_items - dev_total_items) >= .01
diff --git a/integration_tests/tests/consistency/consistency_ad_group_report.sql b/integration_tests/tests/consistency/consistency_ad_group_report.sql
@@ -8,11 +8,11 @@ with prod as (
         ad_group_id,
         sum(clicks) as clicks, 
         sum(impressions) as impressions,
-        sum(spend) as spend
-        {# sum(conversions) as conversions,
+        sum(spend) as spend,
+        sum(conversions) as conversions,
         sum(view_through_conversions) as view_through_conversions,
         sum(total_value) as total_value,
-        sum(total_items) as total_items #}
+        sum(total_items) as total_items
     from {{ target.schema }}_reddit_ads_prod.reddit_ads__ad_group_report
     group by 1
 ),
@@ -22,11 +22,11 @@ dev as (
         ad_group_id,
         sum(clicks) as clicks, 
         sum(impressions) as impressions,
-        sum(spend) as spend
-        {# sum(conversions) as conversions,
+        sum(spend) as spend,
+        sum(conversions) as conversions,
         sum(view_through_conversions) as view_through_conversions,
         sum(total_value) as total_value,
-        sum(total_items) as total_items #}
+        sum(total_items) as total_items
     from {{ target.schema }}_reddit_ads_dev.reddit_ads__ad_group_report
     group by 1
 ),
@@ -39,15 +39,15 @@ final as (
         prod.impressions as prod_impressions,
         dev.impressions as dev_impressions,
         prod.spend as prod_spend,
-        dev.spend as dev_spend
-        {# prod.conversions as prod_conversions,
+        dev.spend as dev_spend,
+        prod.conversions as prod_conversions,
         dev.conversions as dev_conversions,
         prod.view_through_conversions as prod_view_through_conversions,
         dev.view_through_conversions as dev_view_through_conversions,
         prod.total_value as prod_total_value,
         dev.total_value as dev_total_value,
         prod.total_items as prod_total_items,
-        dev.total_items as dev_total_items #}
+        dev.total_items as dev_total_items
     from prod
     full outer join dev 
         on dev.ad_group_id = prod.ad_group_id
@@ -59,7 +59,7 @@ where
     abs(prod_clicks - dev_clicks) >= .01
     or abs(prod_impressions - dev_impressions) >= .01
     or abs(prod_spend - dev_spend) >= .01
-    {# or abs(prod_conversions - dev_conversions) >= .01
+    or abs(prod_conversions - dev_conversions) >= .01
     or abs(prod_view_through_conversions - dev_view_through_conversions) >= .01
     or abs(prod_total_value - dev_total_value) >= .01
-    or abs(prod_total_items - dev_total_items) >= .01 #}
+    or abs(prod_total_items - dev_total_items) >= .01
diff --git a/integration_tests/tests/consistency/consistency_ad_report.sql b/integration_tests/tests/consistency/consistency_ad_report.sql
@@ -8,11 +8,11 @@ with prod as (
         ad_id,
         sum(clicks) as clicks, 
         sum(impressions) as impressions,
-        sum(spend) as spend
-        {# sum(conversions) as conversions,
+        sum(spend) as spend,
+        sum(conversions) as conversions,
         sum(view_through_conversions) as view_through_conversions,
         sum(total_value) as total_value,
-        sum(total_items) as total_items #}
+        sum(total_items) as total_items
     from {{ target.schema }}_reddit_ads_prod.reddit_ads__ad_report
     group by 1
 ),
@@ -22,11 +22,11 @@ dev as (
         ad_id,
         sum(clicks) as clicks, 
         sum(impressions) as impressions,
-        sum(spend) as spend
-        {# sum(conversions) as conversions,
+        sum(spend) as spend,
+        sum(conversions) as conversions,
         sum(view_through_conversions) as view_through_conversions,
         sum(total_value) as total_value,
-        sum(total_items) as total_items #}
+        sum(total_items) as total_items
     from {{ target.schema }}_reddit_ads_dev.reddit_ads__ad_report
     group by 1
 ),
@@ -39,15 +39,15 @@ final as (
         prod.impressions as prod_impressions,
         dev.impressions as dev_impressions,
         prod.spend as prod_spend,
-        dev.spend as dev_spend
-        {# prod.conversions as prod_conversions,
+        dev.spend as dev_spend,
+        prod.conversions as prod_conversions,
         dev.conversions as dev_conversions,
         prod.view_through_conversions as prod_view_through_conversions,
         dev.view_through_conversions as dev_view_through_conversions,
         prod.total_value as prod_total_value,
         dev.total_value as dev_total_value,
         prod.total_items as prod_total_items,
-        dev.total_items as dev_total_items #}
+        dev.total_items as dev_total_items
     from prod
     full outer join dev 
         on dev.ad_id = prod.ad_id
@@ -59,7 +59,7 @@ where
     abs(prod_clicks - dev_clicks) >= .01
     or abs(prod_impressions - dev_impressions) >= .01
     or abs(prod_spend - dev_spend) >= .01
-    {# or abs(prod_conversions - dev_conversions) >= .01
+    or abs(prod_conversions - dev_conversions) >= .01
     or abs(prod_view_through_conversions - dev_view_through_conversions) >= .01
     or abs(prod_total_value - dev_total_value) >= .01
-    or abs(prod_total_items - dev_total_items) >= .01 #}
+    or abs(prod_total_items - dev_total_items) >= .01
diff --git a/integration_tests/tests/consistency/consistency_campaign_report.sql b/integration_tests/tests/consistency/consistency_campaign_report.sql
@@ -8,11 +8,11 @@ with prod as (
         campaign_id,
         sum(clicks) as clicks, 
         sum(impressions) as impressions,
-        sum(spend) as spend
-        {# sum(conversions) as conversions,
+        sum(spend) as spend,
+        sum(conversions) as conversions,
         sum(view_through_conversions) as view_through_conversions,
         sum(total_value) as total_value,
-        sum(total_items) as total_items #}
+        sum(total_items) as total_items
     from {{ target.schema }}_reddit_ads_prod.reddit_ads__campaign_report
     group by 1
 ),
@@ -22,11 +22,11 @@ dev as (
         campaign_id,
         sum(clicks) as clicks, 
         sum(impressions) as impressions,
-        sum(spend) as spend
-        {# sum(conversions) as conversions,
+        sum(spend) as spend,
+        sum(conversions) as conversions,
         sum(view_through_conversions) as view_through_conversions,
         sum(total_value) as total_value,
-        sum(total_items) as total_items #}
+        sum(total_items) as total_items
     from {{ target.schema }}_reddit_ads_dev.reddit_ads__campaign_report
     group by 1
 ),
@@ -39,15 +39,15 @@ final as (
         prod.impressions as prod_impressions,
         dev.impressions as dev_impressions,
         prod.spend as prod_spend,
-        dev.spend as dev_spend
-        {# prod.conversions as prod_conversions,
+        dev.spend as dev_spend,
+        prod.conversions as prod_conversions,
         dev.conversions as dev_conversions,
         prod.view_through_conversions as prod_view_through_conversions,
         dev.view_through_conversions as dev_view_through_conversions,
         prod.total_value as prod_total_value,
         dev.total_value as dev_total_value,
         prod.total_items as prod_total_items,
-        dev.total_items as dev_total_items #}
+        dev.total_items as dev_total_items
     from prod
     full outer join dev 
         on dev.campaign_id = prod.campaign_id
@@ -59,7 +59,7 @@ where
     abs(prod_clicks - dev_clicks) >= .01
     or abs(prod_impressions - dev_impressions) >= .01
     or abs(prod_spend - dev_spend) >= .01
-    {# or abs(prod_conversions - dev_conversions) >= .01
+    or abs(prod_conversions - dev_conversions) >= .01
     or abs(prod_view_through_conversions - dev_view_through_conversions) >= .01
     or abs(prod_total_value - dev_total_value) >= .01
-    or abs(prod_total_items - dev_total_items) >= .01 #}
+    or abs(prod_total_items - dev_total_items) >= .01
diff --git a/integration_tests/tests/consistency/consistency_url_report.sql b/integration_tests/tests/consistency/consistency_url_report.sql
@@ -8,11 +8,11 @@ with prod as (
         click_url,
         sum(clicks) as clicks, 
         sum(impressions) as impressions,
-        sum(spend) as spend
-        {# sum(conversions) as conversions,
+        sum(spend) as spend,
+        sum(conversions) as conversions,
         sum(view_through_conversions) as view_through_conversions,
         sum(total_value) as total_value,
-        sum(total_items) as total_items #}
+        sum(total_items) as total_items
     from {{ target.schema }}_reddit_ads_prod.reddit_ads__url_report
     group by 1
 ),
@@ -22,11 +22,11 @@ dev as (
         click_url,
         sum(clicks) as clicks, 
         sum(impressions) as impressions,
-        sum(spend) as spend
-        {# sum(conversions) as conversions,
+        sum(spend) as spend,
+        sum(conversions) as conversions,
         sum(view_through_conversions) as view_through_conversions,
         sum(total_value) as total_value,
-        sum(total_items) as total_items #}
+        sum(total_items) as total_items
     from {{ target.schema }}_reddit_ads_dev.reddit_ads__url_report
     group by 1
 ),
@@ -39,15 +39,15 @@ final as (
         prod.impressions as prod_impressions,
         dev.impressions as dev_impressions,
         prod.spend as prod_spend,
-        dev.spend as dev_spend
-        {# prod.conversions as prod_conversions,
+        dev.spend as dev_spend,
+        prod.conversions as prod_conversions,
         dev.conversions as dev_conversions,
         prod.view_through_conversions as prod_view_through_conversions,
         dev.view_through_conversions as dev_view_through_conversions,
         prod.total_value as prod_total_value,
         dev.total_value as dev_total_value,
         prod.total_items as prod_total_items,
-        dev.total_items as dev_total_items #}
+        dev.total_items as dev_total_items
     from prod
     full outer join dev 
         on dev.click_url = prod.click_url
@@ -59,7 +59,7 @@ where
     abs(prod_clicks - dev_clicks) >= .01
     or abs(prod_impressions - dev_impressions) >= .01
     or abs(prod_spend - dev_spend) >= .01
-    {# or abs(prod_conversions - dev_conversions) >= .01
+    or abs(prod_conversions - dev_conversions) >= .01
     or abs(prod_view_through_conversions - dev_view_through_conversions) >= .01
     or abs(prod_total_value - dev_total_value) >= .01
-    or abs(prod_total_items - dev_total_items) >= .01 #}
+    or abs(prod_total_items - dev_total_items) >= .01