From 178d90545bb07b3d04bc9e47503b698630c9cb20 Mon Sep 17 00:00:00 2001 From: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> Date: Thu, 5 Sep 2024 19:14:48 -0500 Subject: [PATCH] Documentation Standard Updates (#140) * MagicBot/documentation-updates * Apply suggestions from code review * Update README.md --------- Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug-report.yml | 21 +++- .github/ISSUE_TEMPLATE/config.yml | 3 + .github/ISSUE_TEMPLATE/feature-request.yml | 11 +- .../maintainer_pull_request_template.md | 1 - .github/pull_request_template.md | 8 +- .quickstart/quickstart.yml | 10 ++ README.md | 100 +++++++++--------- 7 files changed, 99 insertions(+), 55 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml index aaa74fa5..38c56bc8 100644 --- a/.github/ISSUE_TEMPLATE/bug-report.yml +++ b/.github/ISSUE_TEMPLATE/bug-report.yml @@ -35,6 +35,12 @@ body: description: A concise description of what you expected to happen. validations: required: true + - type: textarea + attributes: + label: Possible solution + description: Were you able to investigate and/or discover a potential fix to this bug in your investigation? If so, it would be much appreciated if you could submit code samples to show us how your fix resolved this issue. + validations: + required: false - type: textarea attributes: label: dbt Project configurations @@ -61,6 +67,19 @@ body: - other (mention it in "Additional Context") validations: required: true + - type: dropdown + id: orchestration_type + attributes: + label: How are you running this dbt package? + multiple: true + options: + - Fivetran Quickstart Data Model + - Fivetran Transformations + - dbt Core™ + - dbt Cloud™ + - other (mention it in "Additional Context") + validations: + required: true - type: textarea attributes: label: dbt Version @@ -83,6 +102,6 @@ body: description: Our team will assess this issue and let you know if we will add it to a future sprint. However, if you would like to expedite the solution, we encourage you to contribute to the package via a PR. Our team will then work with you to approve and merge your contributions as soon as possible. options: - label: Yes. - - label: Yes, but I will probably need assistance. + - label: Yes, but I will need assistance. - label: No. required: false \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index dc85da95..496b3bd6 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -1,4 +1,7 @@ contact_links: + - name: Provide feedback to our dbt package team + url: https://www.surveymonkey.com/r/DQ7K7WW + about: Fill out our survey form to provide valuable feedback to the Fivetran team developing and maintaining the dbt packages. - name: Fivetran connector question url: https://support.fivetran.com/hc about: Have a question about your connector? Check out the Fivetran support portal for more details. diff --git a/.github/ISSUE_TEMPLATE/feature-request.yml b/.github/ISSUE_TEMPLATE/feature-request.yml index 59e71ff8..529e9bc5 100644 --- a/.github/ISSUE_TEMPLATE/feature-request.yml +++ b/.github/ISSUE_TEMPLATE/feature-request.yml @@ -20,6 +20,13 @@ body: description: A clear and concise description of what you want to happen and why you want the new feature. validations: required: true + - type: textarea + attributes: + label: How would you implement this feature? + description: | + How would you build out this feature with your existing data? Any code examples you can provide to help accelerate development on this issue? + validations: + required: true - type: textarea attributes: label: Describe alternatives you've considered @@ -33,8 +40,8 @@ body: label: Are you interested in contributing this feature? description: Our team will assess this feature and let you know if we will add it to a future sprint. However, if you would like to expedite the feature, we encourage you to contribute to the package via a PR. Our team will then work with you to approve and merge your contributions as soon as possible. options: - - label: Yes. - - label: Yes, but I will probably need assistance. + - label: Yes. + - label: Yes, but I will need assistance. - label: No. required: false - type: textarea diff --git a/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md index 1e22b099..32206744 100644 --- a/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md +++ b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md @@ -16,7 +16,6 @@ Please acknowledge that you have successfully performed the following commands l Before marking this PR as "ready for review" the following have been applied: - [ ] The appropriate issue has been linked, tagged, and properly assigned - [ ] All necessary documentation and version upgrades have been applied - - [ ] docs were regenerated (unless this PR does not include any code or yml updates) - [ ] BuildKite integration tests are passing - [ ] Detailed validation steps have been provided below diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index b4e7e8e0..30849fd0 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -19,7 +19,13 @@ - [ ] Yes -**Provide an emoji that best describes your current mood** +**Typically there are additional maintenance changes required before this will be ready for an upcoming release. Are you comfortable with the Fivetran team making a few commits directly to your branch?** + + +- [ ] Yes +- [ ] No + +**If you had to summarize this PR in an emoji, which would it be?** :dancer: diff --git a/.quickstart/quickstart.yml b/.quickstart/quickstart.yml index 498b1479..cf898de9 100644 --- a/.quickstart/quickstart.yml +++ b/.quickstart/quickstart.yml @@ -60,3 +60,13 @@ destination_configurations: dispatch: - macro_namespace: dbt_utils search_order: [ 'spark_utils', 'dbt_utils' ] + +public_models: [ + "quickbooks__balance_sheet", + "quickbooks__general_ledger_by_period", + "quickbooks__general_ledger", + "quickbooks__profit_and_loss", + "quickbooks__expenses_sales_enhanced", + "quickbooks__ap_ar_enhanced", + "quickbooks__cash_flow_statement" +] diff --git a/README.md b/README.md index e6ecfec7..06e909ee 100644 --- a/README.md +++ b/README.md @@ -13,20 +13,20 @@
-# QuickBooks dbt Package ([Docs](https://fivetran.github.io/dbt_quickbooks/)) +# QuickBooks dbt Package ([Docs](https://fivetran.github.io/dbt_quickbooks/)) -# 📖 Table of Contents -- [📣 What does this dbt package do?](https://github.com/fivetran/dbt_quickbooks/#-what-does-this-dbt-package-do) -- [🎯 How do I use the dbt package?](https://github.com/fivetran/dbt_quickbooks_source/#-how-do-i-use-the-dbt-package) +## Table of Contents +- [What does this dbt package do?](https://github.com/fivetran/dbt_quickbooks/#-what-does-this-dbt-package-do) +- [How do I use the dbt package?](https://github.com/fivetran/dbt_quickbooks_source/#-how-do-i-use-the-dbt-package) - [Required steps](https://github.com/fivetran/dbt_quickbooks/#step-1-prerequisites) - [Additional options](https://github.com/fivetran/dbt_quickbooks/#optional-step-5-additional-configurations) -- [🔍 Does this package have dependencies?](https://github.com/fivetran/dbt_quickbooks/#-does-this-package-have-dependencies) -- [🙌 How is this package maintained and can I contribute?](https://github.com/fivetran/dbt_quickbooks/#-how-is-this-package-maintained-and-can-i-contribute) - - [Package Maintenance](https://github.com/fivetran/dbt_quickbooks/#package-maintenance) - - [Contributions](https://github.com/fivetran/dbt_quickbooks/#contributions) -- [🏪 Are there any resources available?](https://github.com/fivetran/dbt_quickbooks/#-are-there-any-resources-available) + - [Does this package have dependencies?](https://github.com/fivetran/dbt_quickbooks/#-does-this-package-have-dependencies) + - [How is this package maintained and can I contribute?](https://github.com/fivetran/dbt_quickbooks/#-how-is-this-package-maintained-and-can-i-contribute) +- [Package Maintenance](https://github.com/fivetran/dbt_quickbooks/#package-maintenance) +- [Contributions](https://github.com/fivetran/dbt_quickbooks/#contributions) +- [Are there any resources available?](https://github.com/fivetran/dbt_quickbooks/#-are-there-any-resources-available) -# 📣 What does this dbt package do? +## What does this dbt package do? - Produces modeled tables that leverage QuickBooks data from [Fivetran's connector](https://fivetran.com/docs/applications/quickbooks) in the format described by [this ERD](https://fivetran.com/docs/applications/quickbooks#schemainformation) and builds off the output of our [QuickBooks source package](https://github.com/fivetran/dbt_quickbooks_source). - Enables users with insights into their QuickBooks data that can be used for financial statement reporting and deeper analysis. The package achieves this by: @@ -38,11 +38,11 @@ - Generates a comprehensive data dictionary of your source and modeled QuickBooks data through the [dbt docs site](https://fivetran.github.io/dbt_quickbooks/). -The following table provides a detailed list of all models materialized within this package by default. A dependency on the source package is declared in this package's `packages.yml` file, so it will automatically download when you run `dbt deps`. The primary outputs of this package are described below. Intermediate models are used to create these output models. +The following table provides a detailed list of all tables materialized within this package by default. The primary outputs of this package are described below. Intermediate tables are used to create these outputs. -> TIP: See more details about these models in the package's [dbt docs site](https://fivetran.github.io/dbt_quickbooks/#!/overview?g_v=1&g_e=seeds). +> TIP: See more details about these tables in the package's [dbt docs site](https://fivetran.github.io/dbt_quickbooks/#!/overview?g_v=1&g_e=seeds). -| **Model** | **Description** | +| **Table** | **Description** | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | | [quickbooks__general_ledger](https://fivetran.github.io/dbt_quickbooks/#!/model/model.quickbooks.quickbooks__general_ledger) | Table containing a comprehensive list of all transactions with offsetting debit and credit entries to accounts. | | [quickbooks__general_ledger_by_period](https://fivetran.github.io/dbt_quickbooks/#!/model/model.quickbooks.quickbooks__general_ledger_by_period) | Table containing the beginning balance, ending balance, and net change of the dollar amount for each month since the first transaction. This table can be used to generate a balance sheet and income statement for your business. | @@ -53,20 +53,20 @@ The following table provides a detailed list of all models materialized within t | [quickbooks__expenses_sales_enhanced](https://fivetran.github.io/dbt_quickbooks/#!/model/model.quickbooks.quickbooks__expenses_sales_enhanced) | Table providing enhanced customer, vendor, and account details for each expense and sale transaction. | -## Multicurrency Support (And Existing Limitations) +### Multicurrency Support (And Existing Limitations) > [dbt_quickbooks](https://github.com/fivetran/dbt_quickbooks) and [dbt_quickbooks_source](https://github.com/fivetran/dbt_quickbooks_source) now supports multicurrency by bringing in values by specifying `*_converted_*` values for cash amounts. More details are [available in the DECISIONLOG](https://github.com/fivetran/dbt_quickbooks/blob/main/DECISIONLOG.md#multicurrency-vs-single-currency-configuration). There are specific limitations if you need converted amounts for credit card payments and transfers, as we do not currently have the ability to validate a best approach to convert the original amounts into their proper home currency. [Please open an issue with us](https://github.com/fivetran/dbt_quickbooks/issues/new/choose) if this is critical and/or you believe you can help contribute to scoping out adding that functionality in. -# 🎯 How do I use the dbt package? -## Step 1: Prerequisites +## How do I use the dbt package? +### Step 1: Prerequisites To use this dbt package, you must have the following: - At least one Fivetran QuickBooks connector syncing data into your destination. - A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination. -## Step 2: Install the package +### Step 2: Install the package Include the following QuickBooks package version in your `packages.yml` file. > TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages. @@ -79,7 +79,7 @@ packages: Do NOT include the `quickbooks_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well. -## Step 3: Define database and schema variables +### Step 3: Define database and schema variables By default, this package runs using your destination and the `quickbooks` schema of your [target database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile). If this is not where your QuickBooks data is (for example, if your QuickBooks schema is named `quickbooks_fivetran`), add the following configuration to your root `dbt_project.yml` file: ```yml @@ -88,7 +88,7 @@ vars: quickbooks_schema: your_schema_name ``` -## Step 4: Enabling/Disabling Models +### Step 4: Enabling/Disabling Models Your QuickBooks connector might not sync every table that this package expects. This package takes into consideration that not every QuickBooks account utilizes the same transactional tables. By default, most variables' values are assumed to be `true` (with exception of `using_credit_card_payment_txn` and `using_purchase_order`). In other to enable or disable the relevant functionality in the package, you will need to add the relevant variables: @@ -113,9 +113,9 @@ vars: using_purchase_order: true #enable if you want to include purchase orders in your staging models ``` -## (Optional) Step 5: Additional Configurations +### (Optional) Step 5: Additional Configurations -### Unioning Multiple Quickbooks Connectors +#### Unioning Multiple Quickbooks Connectors If you have multiple Quickbooks connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `quickbooks_union_schemas` or `quickbooks_union_databases` variables: ```yml @@ -129,8 +129,8 @@ vars: quickbooks_union_databases: ['quickbooks_usa','quickbooks_canada'] # use this if the data is in different databases/projects but uses the same schema name ``` -### Configuring Account Type Names -Within a few of the double entry models in this package a mapping takes place to assign certain transaction type's debits/credits to the appropriate offset account (ie. Accounts Payable, Accounts Receivable, Undeposited Funds, and SalesOfProductIncome) reference. While our current filtered logic within our intermediate models account for the default values, it's possible your use case relies on different account types to reference. +#### Configuring Account Type Names +Within a few of the double entry models in this package a mapping takes place to assign certain transaction type's debits/credits to the appropriate offset account (ie. Accounts Payable, Accounts Receivable, Undeposited Funds, and SalesOfProductIncome) reference. While our current filtered logic within our intermediate models account for the default values, it's possible your use case relies on different account types to reference. If you have a different value to reference for each type, you will need to configure the `account_type` and `account_sub_type` variables that account for these variables in your `dbt_project.yml`. @@ -142,16 +142,16 @@ vars: quickbooks__sales_of_product_income_reference: account_sales_of_product_income_value # 'SalesOfProductIncome' is the default filter set for the account_subtype reference. ``` -### Customize the Cash Flow Model +#### Customize the Cash Flow Model **IMPORTANT**: It is very likely you will need to reconfigure your `cash_flow_type` to make sure your cash flow statement matches your specific use case. Please examine the following instructions. The current default numbering for ordinals and default cash flow types are set in [the `int_quickbooks__cash_flow_classifications`](https://github.com/fivetran/dbt_quickbooks/blob/main/models/intermediate/int_quickbooks__cash_flow_classifications.sql) model. It's based on best practices for cash flow statements leveraging the indirect method in accounting. You can see these ordinals being created in the `int_quickbooks__cash_flow_classifications` model, then implemented in the `quickbooks__cash_flow_statement` model. The `cash_flow_type` value is assigned off of `account_class`, `account_name` or `account_type`, and the cash flow ordinal is assigned off of `cash_flow_type`. If you'd like to modify either of these configurations, take the following steps to configure the fields you'd like to modify: -1) Create a csv file within your root (not the dbt package) `seeds` folder, then configure your `cash_flow_statement_type_ordinal` variable in your `dbt_project.yml` to reference the seed file name. +1) Create a csv file within your root (not the dbt package) `seeds` folder, then configure your `cash_flow_statement_type_ordinal` variable in your `dbt_project.yml` to reference the seed file name. - For example, if you created a seed file named `quickbooks_cash_flow_types_ordinals.csv`, then you would edit the `cash_flow_statement_type_ordinal` in your root `dbt_project.yml` as such. - + ```yml vars: cash_flow_statement_type_ordinal: "{{ ref('quickbooks_cash_flow_types_ordinals') }}" @@ -165,7 +165,7 @@ These are our recommended best practices to follow with your seed file (you can - In `cash_flow_statement_type_ordinal_example`, we recommend creating ordinals for each `cash_flow_type` value available (the default types are `Cash or Cash Equivalents`, `Operating`, `Investing`, `Financing` as per best financial practices, but you can configure as you like in your seed file) to make sure each cash flow statement type can be easily ordered. Then you can create any additional customization as needed with the more specific account fields to order even further. - In `cash_flow_statement_type_ordinal_example`, the `report` field should always be `Cash Flow`. -We'd love for you to share your experiences with the cash flow seed file with us [in the Fivetran community user group](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) so we can make these model and seed configurations even better for you in the future! +We'd love for you to share your experiences with the cash flow seed file with us [in the Fivetran community user group](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) so we can make these model and seed configurations even better for you in the future!. ### Customize the account ordering of your financial models. [The current default numbering for ordinals](https://github.com/fivetran/dbt_quickbooks/blob/main/models/quickbooks__general_ledger_by_period.sql#L44-L50) is based on best practices for balance sheets and profit-and-loss statements in accounting. You can see these ordinals in action in the `quickbooks__general_ledger_by_period`, `quickbooks__balance_sheet` and `quickbooks__profit_and_loss` models. The ordinals are assigned off of the `account_class` values. @@ -176,22 +176,22 @@ If you'd like to modify this, take the following steps: - For example, if you created a seed file named `quickbooks_ordinals.csv`, then you would edit the `financial_statement_ordinal` in your root `dbt_project.yml` as such. ```yml - vars: +vars: financial_statement_ordinal: "{{ ref('quickbooks_ordinals') }}" - -2) Examine the [`financial_statement_ordinal_example` file](https://github.com/fivetran/dbt_quickbooks/blob/main/example_ordinal_seeds/financial_statement_ordinal_example.csv) to see what your sample seed file should look like. (NOTE: Make sure that your `seed` file name is different from `financial_statement_ordinal_example` to avoid errors.). You can use this file as an example and follow the steps in (1) to see what the ordering of the data looks like, then modify as needed. + +2) Examine the [`financial_statement_ordinal_example` file](https://github.com/fivetran/dbt_quickbooks/blob/main/example_ordinal_seeds/financial_statement_ordinal_example.csv) to see what your sample seed file should look like. (NOTE: Make sure that your `seed` file name is different from `financial_statement_ordinal_example` to avoid errors.). You can use this file as an example and follow the steps in (1) to see what the ordering of the data looks like, then modify as needed. 3) When adding and making changes to the seed file, you will need to run the `dbt build` command to compile the updated seed data into the above financial reporting models. -These are our recommended best practices to follow with your seed file ([you can see them in action in the `financial_statement_ordinal_example` file](https://github.com/fivetran/dbt_quickbooks/blob/main/example_ordinal_seeds/financial_statement_ordinal_example.csv)): -- REQUIRED: In each row of the seed file, only populate **ONE** of the `account_class`, `account_type`, `account_sub_type`, and `account_number` columns to avoid duplicated ordinals and test failures. This should also make the logic cleaner in defining which account value takes precedence in the ordering hierarchy. -- We recommend creating ordinals for each `account_class` value available (usually 'Asset', 'Liability', 'Equity' for the Profit and Loss sheet, and 'Revenue' and 'Expense' for the Balance Sheet) to make sure each financial reporting line has an ordinal assigned to it. Then you can create any additional customization as needed with the more specific account fields to order even further. -- Fill out the `report` field as either `Balance Sheet` if the particular row belongs in `quickbooks__balance_sheet`, or `Profit and Loss` for `quickbooks__profit_and_loss`. -- We recommend ordering the `ordinal` for each report separately in the seed, i.e. have ordinals for `quickbooks__balance_sheet` and `quickbooks__profit_and_loss` start at 1 each, to make your reporting more clean. +These are our recommended best practices to follow with your seed file ([you can see them in action in the `financial_statement_ordinal_example` file](https://github.com/fivetran/dbt_quickbooks/blob/main/example_ordinal_seeds/financial_statement_ordinal_example.csv)): +- REQUIRED: In each row of the seed file, only populate **ONE** of the `account_class`, `account_type`, `account_sub_type`, and `account_number` columns to avoid duplicated ordinals and test failures. This should also make the logic cleaner in defining which account value takes precedence in the ordering hierarchy. +- We recommend creating ordinals for each `account_class` value available (usually 'Asset', 'Liability', 'Equity' for the Profit and Loss sheet, and 'Revenue' and 'Expense' for the Balance Sheet) to make sure each financial reporting line has an ordinal assigned to it. Then you can create any additional customization as needed with the more specific account fields to order even further. +- Fill out the `report` field as either `Balance Sheet` if the particular row belongs in `quickbooks__balance_sheet`, or `Profit and Loss` for `quickbooks__profit_and_loss`. +- We recommend ordering the `ordinal` for each report separately in the seed, i.e. have ordinals for `quickbooks__balance_sheet` and `quickbooks__profit_and_loss` start at 1 each, to make your reporting more clean. -We'd love for you to share your experiences with the ordinal seed file with us [in the Fivetran community user group](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) so we can make these model and seed configurations even better for you in the future! +We'd love for you to share your experiences with the ordinal seed file with us [in the Fivetran community user group](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) so we can make these model and seed configurations even better for you in the future. -### Changing the Build Schema +#### Changing the Build Schema By default this package will build the QuickBooks staging models within a schema titled (