Proposal for a New refactor Feature in Dataform CLI

[hiracky16]

Overview

I would like to propose adding a new refactor command to the Dataform CLI. This feature would help synchronize SQLX files with actual BigQuery tables and propagate column descriptions from tables specified in the from clause.

Details

• I have developed a project called dataform-osmosis, which implements functionality to inherit column information (such as column names and descriptions) from from tables in SQLX files.
• I believe this functionality can be beneficial as a built-in feature of Dataform CLI.

Proposed Functionality

The dataform refactor command would provide the following features:

1. Automatically synchronize SQLX files with column information from actual BigQuery tables.
2. Inherit column descriptions from tables referenced in the from clause and propagate them to the SQLX files.
3. Ensure column ordering and descriptions are consistent between SQLX files and the underlying tables.

Benefits

• Improved Consistency: Automates the synchronization of SQLX files with actual tables, reducing manual effort and potential errors.
• Enhanced Efficiency: Teams can maintain table definitions more easily, making schema management smoother.
  Example Use Case

Example Use Case

dataform refactor definitions/example.sqlx

Request for Feedback

I would love to hear your thoughts on whether this feature would align with the goals of Dataform and any considerations or improvements you'd suggest for its implementation.