Skip to content

Commit

Permalink
Merge branch 'current' into mirnawong1-patch-5
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Jul 19, 2023
2 parents 45d2c19 + 1991ca0 commit f770ae9
Show file tree
Hide file tree
Showing 23 changed files with 253 additions and 84 deletions.
8 changes: 6 additions & 2 deletions website/dbt-versions.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
exports.versions = [
{
version: "1.6",
EOLDate: "2024-07-20", // placeholder - need to confirm the final date
EOLDate: "2024-07-31",
isPrerelease: true
},
{
Expand Down Expand Up @@ -31,6 +31,10 @@ exports.versions = [
]

exports.versionedPages = [
{
"page": "reference/commands/clone",
"firstVersion": "1.6",
},
{
"page": "docs/collaborate/govern/project-dependencies",
"firstVersion": "1.6",
Expand All @@ -55,7 +59,7 @@ exports.versionedPages = [
"page": "docs/collaborate/govern/model-contracts",
"firstVersion": "1.5",
},
{
{
"page": "reference/commands/show",
"firstVersion": "1.5",
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ control (RBAC).

The following permission sets are available for assignment in dbt Cloud Enterprise accounts. They
can be granted to dbt Cloud groups which are then in turn granted to users. A dbt Cloud group
can be associated with more than one permission sets.
can be associated with more than one permission set.

### Account Admin

Expand Down Expand Up @@ -72,13 +72,13 @@ Billing Admins have access to modify certain account-level settings related to b
- **Has permissions on:** Authorized projects, account-level settings
- **License restrictions:** must have a developer license

Project Creators have write and read-only access to dbt Cloud accounts, but do not have the permissions required to modify SSO settings and account integrations.
Project Creators can access, create, or modify projects and other settings in dbt Cloud. However, they don't have permission to modify SSO settings or account integrations.

Users with Project Creator permissions can:

- View Account Settings
- View and modify project users
- Create, delete and modify all projects in an account
- Create, delete, and modify all projects in an account
- Create, delete, and modify Connections
- Create, delete, and modify Environments
- Create, delete, and modify Jobs
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ Client Secret for use in dbt Cloud.
| **Application type** | internal | required |
| **Application name** | dbt Cloud | required |
| **Application logo** | Download the logo <a href="https://www.getdbt.com/ui/img/dbt-icon.png" target="_blank" rel="noopener noreferrer">here</a> | optional |
| **Authorized domains** | `getdbt.com` | If deploying into a VPC, use the domain for your deployment |
| **Authorized domains** | `getdbt.com` (US) `dbt.com` (EMEA or AU) | If deploying into a VPC, use the domain for your deployment |
| **Scopes** | `email, profile, openid` | The default scopes are sufficient |

<Lightbox src="/img/docs/dbt-cloud/dbt-cloud-enterprise/gsuite/gsuite-sso-consent-top.png" title="GSuite Consent Screen configuration"/>
Expand Down
6 changes: 3 additions & 3 deletions website/docs/docs/collaborate/govern/model-versions.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Instead, for mature models at larger organizations, powering queries inside & ou

During that migration window, anywhere that model is being used downstream, it can continue to be referenced at a specific version.

In the future, dbt will also offer first-class support for **deprecating models** ([dbt-core#7433](https://github.com/dbt-labs/dbt-core/issues/7433)). Taken together, model versions and deprecation offer a pathway for model producers to _sunset_ old models, and consumers the time to _migrate_ across breaking changes. It's a way of managing change across an organization: develop a new version, bump the latest, slate the old version for deprecation, update downstream references, and then remove the old version.
dbt Core 1.6 introduced first-class support for **deprecating models** by specifying a [`deprecation_date`](/reference/resource-properties/deprecation_date). Taken together, model versions and deprecation offer a pathway for model producers to _sunset_ old models, and consumers the time to _migrate_ across breaking changes. It's a way of managing change across an organization: develop a new version, bump the latest, slate the old version for deprecation, update downstream references, and then remove the old version.

There is a real trade-off that exists here—the cost to frequently migrate downstream code, and the cost (and clutter) of materializing multiple versions of a model in the data warehouse. Model versions do not make that problem go away, but by setting a deprecation date, and communicating a clear window for consumers to gracefully migrate off old versions, they put a known boundary on the cost of that migration.

Expand Down Expand Up @@ -73,7 +73,7 @@ As the **producer** of a versioned model:
- You keep track of all live versions in one place, rather than scattering them throughout the codebase
- You can reuse the model's configuration, and highlight just the diffs between versions
- You can select models to build (or not) based on whether they're a `latest`, `prerelease`, or `old` version
- dbt will notify consumers of your versioned model when new versions become available, or (in the future) when they are slated for deprecation
- dbt will notify consumers of your versioned model when new versions become available, or when they are slated for deprecation

As the **consumer** of a versioned model:
- You use a consistent `ref`, with the option of pinning to a specific live version
Expand Down Expand Up @@ -109,7 +109,7 @@ selectors:
</File>
Because dbt knows that these models are _actually the same model_, it can notify downstream consumers as new versions become available, and (in the future) as older versions are slated for deprecation.
Because dbt knows that these models are _actually the same model_, it can notify downstream consumers as new versions become available, and as older versions are slated for deprecation.
```bash
Found an unpinned reference to versioned model 'dim_customers'.
Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/collaborate/govern/project-dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ sidebar_label: "Project dependencies"
description: "Reference public models across dbt projects"
---

:::info
"Project" dependencies and cross-project `ref` is currently in closed beta and are features of dbt Cloud Enterprise. To access these features, please contact your account team.
:::caution Closed Beta - dbt Cloud Enterprise
"Project" dependencies and cross-project `ref` are features of dbt Cloud Enterprise, currently in Closed Beta. To access these features while they are in beta, please contact your account team at dbt Labs.
:::

For a long time, dbt has supported code reuse and extension by installing other projects as [packages](/docs/build/packages). When you install another project as a package, you are pulling in its full source code, and adding it to your own. This enables you to call macros and run models defined in that other project.
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/dbt-cloud-apis/admin-cloud-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The dbt Cloud Administrative API is enabled by default for [Team and Enterprise
- Manage your dbt Cloud account
- and more

Check out our dbt Cloud Admin API docs to help you access the API:
dbt Cloud currently supports two versions of the Administrative API: v2 and v3. In general, v3 is the recommended version to use, but we don't yet have all our v2 routes upgraded to v3. We're currently working on this. If you can't find something in our v3 docs, check out the shorter list of v2 endpoints because you might find it there.

<div className="grid--2-col">

Expand Down
19 changes: 19 additions & 0 deletions website/docs/docs/dbt-cloud-apis/service-tokens.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@ title: "Service account tokens"
id: "service-tokens"
description: "Service account tokens help you define permissions for securing access to your dbt Cloud account and its projects."
---
:::info Important service account token update

If you have service tokens created on or before July 18, 2023, please read [this important update](/docs/dbt-cloud-apis/service-tokens#service-token-update).

:::

## About service tokens

Expand Down Expand Up @@ -92,3 +97,17 @@ Analyst admin service tokens have all the permissions listed in [Analyst](/docs/

**Stakeholder**<br/>
Stakeholder service tokens have all the permissions listed in [Stakeholder](/docs/cloud/manage-access/enterprise-permissions#stakeholder) on the Enterprise Permissions page.


## Service token update

On July 18, 2023, dbt Labs made critical infrastructure changes to service account tokens. These enhancements improve the security and performance of all tokens created after July 18, 2023. To ensure security best practices are in place, we recommend you rotate your service tokens created before this date.

To rotate your token:
1. Navigate to **Account settings** and click **Service tokens** on the left side pane.
2. Verify the **Created** date for the token is _on or before_ July 18, 2023.
<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/service-token-date.png" title="Service token created date"/>
3. Click **+ New Token** on the top right side of the screen. Ensure the new token has the same permissions as the old one.
4. Copy the new token and replace the old one in your systems. Store it in a safe place, as it will not be available again once the creation screen is closed.
5. Delete the old token in dbt Cloud by clicking the **trash can icon**. _Only take this action after the new token is in place to avoid service disruptions_.

64 changes: 50 additions & 14 deletions website/docs/guides/migration/versions/01-upgrading-to-v1.6.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
---
title: "Upgrading to v1.6 (beta)"
title: "Upgrading to v1.6 (prerelease)"
description: New features and changes in dbt Core v1.6
---

:::warning Beta Functionality
:::warning Prerelease

dbt Core v1.6 is in beta, and the features and functionality on this page are subject to change.
dbt Core v1.6 is available as a release candidate. [Final release is planned for July 31.](https://github.com/dbt-labs/dbt-core/issues/7990)

Test it out, and [let us know](https://github.com/dbt-labs/dbt-core/issues/new/choose) if you run into any issues!

:::

## Resources

- [Changelog](https://github.com/dbt-labs/dbt-core/blob/main/CHANGELOG.md)
- [Changelog](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
- [CLI Installation guide](/docs/core/installation)
- [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
- [Release schedule](https://github.com/dbt-labs/dbt-core/issues/7481)
Expand All @@ -22,24 +24,58 @@ dbt Labs is committed to providing backward compatibility for all versions 1.x,

### Behavior changes

**Coming soon**
- dbt Core v1.6 does not support Python 3.7, which reached End Of Life on June 23. Support Python versions are 3.8, 3.9, 3.10, and 3.11.
- As part of the Semantic layer re-launch (in beta), the spec for `metrics` has changed significantly. Migration guide coming soon: https://github.com/dbt-labs/docs.getdbt.com/pull/3705
- Manifest schema version is now v10, reflecting [TODO] changes

### For consumers of dbt artifacts (metadata)

The [manifest](/reference/artifacts/manifest-json) schema version has updated to `v10`. Specific changes:
- Addition of `semantic_models` and changes to `metrics` attributes
- Addition of `deprecation_date` as a model property
- Addition of `on_configuration_change` as default node configuration (to support materialized views)
- Small type changes to `contracts` and `constraints`
- Manifest `metadata` includes `project_name`

### For maintainers of adapter plugins

For more detailed information and to ask questions, please read and comment on the GH discussion: [dbt-labs/dbt-core#7958](https://github.com/dbt-labs/dbt-core/discussions/7958).

## New and changed documentation

[`dbt retry`](/reference/commands/retry) is a new command that executes the previously run command from the point of failure. This convenient command enables you to continue a failed command without rebuilding all upstream dependencies.
### Materialized views

**Materialized view** support (for model and project configs) has been added for three data warehouses:
- [Bigquery](/reference/resource-configs/bigquery-configs#materialized-view)
- [Postgres](/reference/resource-configs/postgres-configs#materialized-view)
- [Redshift](/reference/resource-configs/redshift-configs#materialized-view)
Supported on:
- [Postgres](/reference/resource-configs/postgres-configs#materialized-view)
- [Redshift](/reference/resource-configs/redshift-configs#materialized-view)
- Snowflake (docs forthcoming)

[**Namespacing:**](/faqs/Models/unique-model-names) Model names can be duplicated across different namespaces (packages/projects), so long as they are unique within each package/project. We strongly encourage using [two-argument `ref`](/reference/dbt-jinja-functions/ref#two-argument-variant) when referencing a model from a different package/project.
Support for BigQuery and Databricks forthcoming.

[**Project dependencies**](/docs/collaborate/govern/project-dependencies): Introduces `dependencies.yml` and dependent `projects` as a feature of dbt Cloud Enterprise. Allows enforcing model access (public vs. protected/private) across project/package boundaries. Enables cross-project `ref` of public models, without requiring the installation of upstream source code.
### New commands for mature deployment

### Quick hits
[`dbt retry`](/reference/commands/retry) executes the previously run command from the point of failure. Rebuild just the nodes that errored or skipped in a previous run/build/test, rather than starting over from scratch.

[`dbt clone`](/reference/commands/clone) leverages each data platform's functionality for creating lightweight copies of dbt models from one environment into another. Useful when quickly spinning up a new development environment, or promoting specific models from a staging environment into production.

### Multi-project collaboration

More consistency and flexibility around packages! Resources defined in a package will respect variable and global macro definitions within the scope of that package.
[**Deprecation date**](/reference/resource-properties/deprecation_date): Models can declare a deprecation date that will warn model producers and downstream consumers. This enables clear migration windows for versioned models, and provides a mechanism to facilitate removal of immature or little-used models, helping to avoid project bloat.

[Model names](/faqs/Models/unique-model-names) can be duplicated across different namespaces (projects/packages), so long as they are unique within each project/package. We strongly encourage using [two-argument `ref`](/reference/dbt-jinja-functions/ref#two-argument-variant) when referencing a model from a different package/project.

More consistency and flexibility around packages. Resources defined in a package will respect variable and global macro definitions within the scope of that package.
- `vars` defined in a package's `dbt_project.yml` are now available in the resolution order when compiling nodes in that package, though CLI `--vars` and the root project's `vars` will still take precedence. See ["Variable Precedence"](/docs/build/project-variables#variable-precedence) for details.
- `generate_x_name` macros (defining custom rules for database, schema, alias naming) follow the same pattern as other "global" macros for package-scoped overrides. See [macro dispatch](/reference/dbt-jinja-functions/dispatch) for an overview of the patterns that are possible.

:::caution Closed Beta - dbt Cloud Enterprise
[**Project dependencies**](/docs/collaborate/govern/project-dependencies): Introduces `dependencies.yml` and dependent `projects` as a feature of dbt Cloud Enterprise. Allows enforcing model access (public vs. protected/private) across project/package boundaries. Enables cross-project `ref` of public models, without requiring the installation of upstream source code.
:::

### Quick hits

- [`state:unmodified` and `state:old`](/reference/node-selection/methods#the-state-method) for [MECE](https://en.wikipedia.org/wiki/MECE_principle) stateful selection
- [`invocation_args_dict`](/reference/dbt-jinja-functions/flags#invocation_args_dict) includes full `invocation_command` as string
- [`dbt debug --connection`](/reference/commands/debug) to test just the data platform connection specified in a profile
- [`dbt docs generate --empty-catalog`](/reference/commands/cmd-docs) to skip catalog population while generating docs
- [`--defer-state`](/reference/node-selection/defer) enables more-granular control
1 change: 1 addition & 0 deletions website/docs/reference/artifacts/dbt-artifacts.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ All artifacts produced by dbt include a `metadata` dictionary with these propert

In the manifest, the `metadata` may also include:
- `send_anonymous_usage_stats`: Whether this invocation sent [anonymous usage statistics](/reference/global-configs/usage-stats) while executing.
- `project_name`: The `name` defined in the root project's `dbt_project.yml`. (Added in manifest v10 / dbt Core v1.6)
- `project_id`: Project identifier, hashed from `project_name`, sent with anonymous usage stats if enabled.
- `user_id`: User identifier, stored by default in `~/dbt/.user.yml`, sent with anonymous usage stats if enabled.

Expand Down
57 changes: 11 additions & 46 deletions website/docs/reference/artifacts/manifest-json.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,53 +2,18 @@
title: "Manifest JSON file"
sidebar_label: "Manifest"
---
<VersionBlock firstVersion="1.5">

**dbt Core v1.5 produces schema**: [`v9`](https://schemas.getdbt.com/dbt/manifest/v9/index.html)

</VersionBlock>

<VersionBlock firstVersion="1.4" lastVersion="1.4">

**dbt Core v1.4 produces schema**: [`v8`](https://schemas.getdbt.com/dbt/manifest/v8/index.html)

</VersionBlock>

<VersionBlock firstVersion="1.3" lastVersion="1.3">

**dbt Core v1.3 produces schema**: [`v7`](https://schemas.getdbt.com/dbt/manifest/v7/index.html)

</VersionBlock>

<VersionBlock firstVersion="1.2" lastVersion="1.2">

**dbt Core v1.2 produces schema**: [`v6`](https://schemas.getdbt.com/dbt/manifest/v6/index.html)

</VersionBlock>

<VersionBlock firstVersion="1.1" lastVersion="1.1">

**dbt Core v1.1 produces schema**: [`v5`](https://schemas.getdbt.com/dbt/manifest/v5/index.html)

</VersionBlock>

<VersionBlock firstVersion="1.0" lastVersion="1.0">

**dbt Core v1.0 produces schema**: [`v4`](https://schemas.getdbt.com/dbt/manifest/v4/index.html)

</VersionBlock>

<VersionBlock firstVersion="1.5">

**Produced by:** [`build`](/reference/commands/build) [`compile`](/reference/commands/compile) [`docs generate`](/reference/commands/cmd-docs) [`list`](/reference/commands/list) [`seed`](/reference/commands/seed) [`snapshot`](/reference/commands/snapshot) [`source freshness`](/reference/commands/source) [`test`](/reference/commands/test) [`run`](/reference/commands/run) [`run-operation`](/reference/commands/run-operation)

</VersionBlock>

<VersionBlock lastVersion="1.4">

**Produced by:** [`build`](commands/build) [`compile`](commands/compile) [`docs generate`](commands/cmd-docs) [`list`](commands/list) [`parse`](commands/parse) [`run`](commands/run) [`run-operation`](commands/run-operation) [`seed`](commands/seed) [`show`](commands/show) [`snapshot`](commands/snapshot) [`source freshness`](commands/source) [`test`](commands/test)

</VersionBlock>
| dbt Core version | Manifest version |
|------------------|---------------------------------------------------------------|
| v1.6 | [v10](https://schemas.getdbt.com/dbt/manifest/v10/index.html) |
| v1.5 | [v9](https://schemas.getdbt.com/dbt/manifest/v9/index.html) |
| v1.4 | [v8](https://schemas.getdbt.com/dbt/manifest/v8/index.html) |
| v1.3 | [v7](https://schemas.getdbt.com/dbt/manifest/v7/index.html) |
| v1.2 | [v6](https://schemas.getdbt.com/dbt/manifest/v6/index.html) |
| v1.1 | [v5](https://schemas.getdbt.com/dbt/manifest/v5/index.html) |
| v1.0 | [v4](https://schemas.getdbt.com/dbt/manifest/v4/index.html) |

**Produced by:** Any command that parses your project. This includes all commands **except** [`deps`](/reference/commands/deps), [`clean`](/reference/commands/clean), [`debug`](/reference/commands/debug), [`init`](/reference/commands/init)

This single file contains a full representation of your dbt project's resources (models, tests, macros, etc), including all node configurations and resource properties. Even if you're only running some models or tests, all resources will appear in the manifest (unless they are disabled) with most of their properties. (A few node properties, such as `compiled_sql`, only appear for executed nodes.)

Expand Down
Loading

0 comments on commit f770ae9

Please sign in to comment.