Support generation of CTEs in DataflowToSqlQueryPlanConverter
#1521
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
plypaul
commented
Nov 13, 2024
| optimizers=option_set.optimizers, | ||
| ) | ||
|
|
||
| def convert_using_specifics( |
There was a problem hiding this comment.
Need a better name for this one.
courtneyholcomb
approved these changes
Nov 14, 2024
| SqlColumnPrunerOptimizer(), | ||
| SqlRewritingSubQueryReducer(use_column_alias_in_group_bys=use_column_alias_in_group_by), | ||
| SqlTableAliasSimplifier(), | ||
| ) | ||
| allow_cte = True |
There was a problem hiding this comment.
Isn't this an optimizer? Why treat it differently? Did you turn it into an optimizer in a later PR or did I hallucinate that? 🤔
There was a problem hiding this comment.
This one is a little different from the SQL optimizers because it changes how the dataflow plan converts to the SQL plan. The SQL optimizers convert a SQL plan to another SQL plan. Might have been a similar class you saw?
| logger.debug(LazyFormat("Converting to SQL", nodes_to_convert_to_cte=nodes_to_convert_to_cte)) | ||
|
|
||
| if len(nodes_to_convert_to_cte) == 0: | ||
| # Avoid `DataflowNodeToSqlCteVisitor` code path for better isolation during rollout. |
There was a problem hiding this comment.
Thanks for clarifying this 👍
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR:
SqlGenerationOptionSetto encapsulate the options for how SQL should be generated from the dataflow plan.O5level that uses all previous optimizers and generates CTEs.DataflowToSqlQueryPlanConverterto useSqlGenerationOptionSet. When theallow_cteoption it set, converts the common nodes (as implemented in Add a method to figure out common nodes in a dataflow plan #1520) in a dataflow plan to map to a CTE instead of a subquery.Since CTEs are not generated by default, the generated SQL is the same for test cases and there are no snapshot changes (aside from the ones that specifically test this feature).