Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Include Time Spine Nodes in Dataflow Plan #1548

Open
wants to merge 7 commits into
base: court/simp7
Choose a base branch
from
Open

Include Time Spine Nodes in Dataflow Plan #1548

wants to merge 7 commits into from

Conversation

courtneyholcomb
Copy link
Contributor

@courtneyholcomb courtneyholcomb commented Nov 23, 2024 •
edited
Loading

Previously, time spine nodes were built on the fly in the dataflow to SQL logic. This came with several limitations, including:

  • We can't apply where constraints and time constraints using the standard dataflow plan logic. This limitation was the main driver for this PR stack.
  • We can't track time spines as source nodes. We might want to add these to the dbt DAG at some point.

This PR adds logic to build time spine nodes in the dataflow plan, instead.

@courtneyholcomb
Support both MetricTimeDimensionTransformNodes and ReadSqlSourceNodes…
bcd959c 
… for time spines

The metric time nodes are used for resolving metric_time without metrics. The read SQL nodes will be used for time spine joins.
@courtneyholcomb
Add TransformTimeDimensionsNode
f6f44f0
@courtneyholcomb courtneyholcomb changed the base branch from main to court/simp7 November 23, 2024 06:01
@courtneyholcomb
Add logic to build time spine nodes in the dataflow plan
1ec4c63 
This will allow us to stop building time spine nodes on the fly in the dataflow to SQL logic.
@courtneyholcomb courtneyholcomb changed the title Court/simp8 Build Time Spine Nodes in Dataflow Plan Nov 23, 2024
@courtneyholcomb courtneyholcomb changed the title Build Time Spine Nodes in Dataflow Plan Include Time Spine Nodes in Dataflow Plan Nov 23, 2024
@dbt-labs dbt-labs deleted a comment from github-actions bot Nov 23, 2024
@dbt-labs dbt-labs deleted a comment from github-actions bot Nov 23, 2024
@courtneyholcomb courtneyholcomb added the Run Tests With Other SQL Engines Runs the test suite against the SQL engines in our target environment label Nov 23, 2024
@courtneyholcomb courtneyholcomb deployed to DW_INTEGRATION_TESTS November 23, 2024 06:46 — with GitHub Actions Active
@courtneyholcomb courtneyholcomb marked this pull request as ready for review November 23, 2024 06:46
@github-actions github-actions bot removed the Run Tests With Other SQL Engines Runs the test suite against the SQL engines in our target environment label Nov 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@plypaul plypaul Awaiting requested review from plypaul

Assignees
No one assigned
Labels
Skip Changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@courtneyholcomb