Skip to content

Releases: databrickslabs/lsql

v0.14.1

19 Nov 16:04
c5d27d2
Compare
Choose a tag to compare
Release v0.14.1 (#352)

* Changes to work with Databricks SDK `v0.38.0`
([#350](https://github.com/databrickslabs/lsql/issues/350)). In this
release, we have upgraded the Databricks SDK to version 0.38.0 from
version 0.37.0 to ensure compatibility with the latest SDK and address
several issues. The update includes changes to make the code compatible
with the new SDK version, removing the need for `.as_dict()` method
calls when creating or updating dashboards and utilizing a
`sdk_dashboard` variable for interacting with the Databricks workspace.
We also updated the dependencies to "databricks-labs-blueprint[yaml]"
package version greater than or equal to 0.4.2 and `sqlglot` package
version greater than or equal to 22.3.1. The `test_core.py` file has
been updated to address multiple issues
([#349](https://github.com/databrickslabs/lsql/issues/349) to
[#332](https://github.com/databrickslabs/lsql/issues/332)) related to
the Databricks SDK and the `test_dashboards.py` file has been revised to
work with the new SDK version. These changes improve integration with
Databricks' lakeview dashboards, simplify the code, and ensure
compatibility with the latest SDK version, resolving issues
[#349](https://github.com/databrickslabs/lsql/issues/349) to
[#332](https://github.com/databrickslabs/lsql/issues/332).
* Specify the minimum required version of `databricks-sdk` as 0.37.0
([#331](https://github.com/databrickslabs/lsql/issues/331)). In this
release, we have updated the minimum required version of the
`databricks-sdk` package to 0.37.0 from 0.29.0 in the `pyproject.toml`
file to ensure compatibility with the latest version. This change was
made necessary due to updates made in issue
[#320](https://github.com/databrickslabs/lsql/issues/320). To
accommodate any patch release of `databricks-sdk` with a major and minor
version of 0.37, we have updated the dependency constraint to use the
`~=` operator, resolving issue
[#330](https://github.com/databrickslabs/lsql/issues/330). These changes
are intended to enhance the compatibility and stability of our software.

v0.14.0

15 Nov 15:39
@nfx nfx
383bb80
Compare
Choose a tag to compare
  • Added nightly tests run at 4:45am UTC (#318). A new nightly workflow has been added to the codebase, designed to automate a series of jobs every day at 4:45am UTC on the larger environment. The workflow includes permissions for writing id-tokens, accessing issues, reading contents and pull-requests. It checks out the code with a full fetch-depth, installs Python 3.10, and uses hatch 1.9.4. The key step in this workflow is the execution of nightly tests using the databrickslabs/sandbox/acceptance action, which creates issues if necessary. The workflow utilizes several secrets, including VAULT_URI, GITHUB_TOKEN, ARM_CLIENT_ID, and ARM_TENANT_ID, and sets the TEST_NIGHTLY environment variable to true. Additionally, the workflow is part of a concurrency group called "single-acceptance-job-per-repo", ensuring that only one acceptance job runs at a time per repository.
  • Bump codecov/codecov-action from 4 to 5 (#319). In this version update, the Codecov GitHub Action has been upgraded from 4 to 5, bringing improved functionality and new features. This new version utilizes the Codecov Wrapper to encapsulate the CLI, enabling faster updates. Additionally, an opt-out feature has been introduced for tokens in public repositories, allowing contributors and other members to upload coverage reports without requiring access to the Codecov token. The upgrade also includes changes to the arguments: file is now deprecated and replaced with files, and plugin is deprecated and replaced with plugins. New arguments have been added, including binary, gcov_args, gcov_executable, gcov_ignore, gcov_include, report_type, skip_validation, and swift_project. Comprehensive documentation on these changes can be found in the release notes and changelog.
  • Fixed RuntimeBackend exception handling (#328). In this release, we have made significant improvements to the exception handling in the RuntimeBackend component, addressing issues reported in tickets #328, #327, #326, and #325. We have updated the execute and fetch methods to handle exceptions more gracefully and changed exception handling from catching Exception to catching BaseException for more comprehensive error handling. Additionally, we have updated the pyproject.toml file to use a newer version of the databricks-labs-pytester package (0.2.1 to 0.5.0) which may have contributed to the resolution of these issues. Furthermore, the test_backends.py file has been updated to improve the readability and user-friendliness of the test output for the functions testing if a NotFound, BadRequest, or Unknown exception is raised when executing and fetching statements. The test_runtime_backend_use_statements function has also been updated to print PASSED or FAILED instead of returning those values. These changes enhance the robustness of the exception handling mechanism in the RuntimeBackend class and update related unit tests.

Dependency updates:

  • Bump codecov/codecov-action from 4 to 5 (#319).

Contributors: @nfx, @JCZuurmond, @dependabot[bot]

v0.13.0

08 Nov 17:07
@nfx nfx
48c287e
Compare
Choose a tag to compare
  • Added escape_name function to escape individual SQL names and escape_full_name function to escape dot-separated full names (#316). Two new functions, escape_name and escape_full_name, have been added to the databricks.labs.lsql.escapes module for escaping SQL names. The escape_name function takes a single name as an input and returns it enclosed in backticks, while escape_full_name handles dot-separated full names by escaping each individual component. These functions have been ported from the databrickslabs/ucx repository and are designed to provide a consistent way to escape names and full names in SQL statements, improving the robustness of the system by preventing issues caused by unescaped special characters in SQL names. The test suite includes various cases, including single names, full names with different combinations of escaped and unescaped components, and special characters, with a specific focus on the scenario where the column name contains a period.
  • Bump actions/checkout from 4.2.0 to 4.2.1 (#304). In this pull request, the actions/checkout dependency is updated from version 4.2.0 to 4.2.1 in the .github/workflows/release.yml file. This update includes a new feature where refs/* are checked out by commit if provided, falling back to the ref specified by the @orhantoy user. This change improves the flexibility of the action, allowing users to specify a commit or branch for checkout. The pull request also introduces a new contributor, @Jcambass, who added a workflow file for publishing releases to an immutable action package. The commits for this release include changes to prepare for the 4.2.1 release, add a workflow file for publishing releases, and check out other refs/* by commit if provided, falling back to ref. This pull request has been reviewed and approved by Dependabot.
  • Bump actions/checkout from 4.2.1 to 4.2.2 (#310). This is a pull request to update the actions/checkout dependency from version 4.2.1 to 4.2.2, which includes improvements to the url-helper.ts file that now utilize well-known environment variables and expanded unit test coverage for the isGhes function. The actions/checkout action is commonly used in GitHub Actions workflows for checking out a repository at a specific commit or branch. The changes in this update are internal to the actions/checkout action and should not affect the functionality of the project utilizing this action. The pull request also includes details on the commits and compatibility score for the upgrade, and reviewers can manage and merge the request using Dependabot commands once the changes have been verified.
  • Bump databrickslabs/sandbox from acceptance/v0.3.0 to 0.3.1 (#307). In this release, the databrickslabs/sandbox dependency has been updated from version acceptance/v0.3.0 to 0.3.1. This update includes previously tagged commits, bug fixes for git-related libraries, and resolution of the unsupported protocol scheme error. The README has been updated with more information on using the databricks labs sandbox command, and installation instructions have been improved. Additionally, there have been dependency updates for go-git libraries and golang.org/x/crypto in the /go-libs and /runtime-packages directories. New commits in this release allow larger logs from acceptance tests and implement experimental OIDC refresh functionality. Ignore conditions have been applied to prevent conflicts with previous versions of the dependency. This update is recommended for users who want to take advantage of the latest bug fixes and improvements.
  • Bump databrickslabs/sandbox from acceptance/v0.3.1 to 0.4.2 (#315). In this release, the databrickslabs/sandbox dependency has been updated from version acceptance/v0.3.1 to 0.4.2. This update includes bug fixes, dependency updates, and additional go-git libraries. Specifically, the Run integration tests job in the GitHub Actions workflow has been updated to use the new version of the databrickslabs/sandbox/acceptance Docker image. The updated version also includes install instructions, usage instructions in the README, and a modification to provide more git-related libraries. Additionally, there were several updates to dependencies, including golang.org/x/crypto version 0.16.0 to 0.17.0. Dependabot, a tool that manages dependencies in GitHub projects, is responsible for the update and provides instructions for resolving any conflicts or merging the changes into the project. This update is intended to improve the functionality and reliability of the databrickslabs/sandbox dependency.
  • Deprecate Row.as_dict() (#309). In this release, we are introducing a deprecation warning for the as_dict() method in the Row class, which will be removed in favor of the asDict() method. This change aims to maintain consistency with Spark's Row behavior and prevent subtle bugs when switching between different backends. The deprecation warning will be implemented using Python's warnings mechanism, including the new annotation in Python 3.13 for static code analysis. The existing functionality of fetching values from the database through StatementExecutionExt remains unchanged. We recommend that clients update their code to use .asDict() instead of .as_dict() to avoid any disruptions. A new test case test_row_as_dict_deprecated() has been added to verify the deprecation warning for Row.as_dict().
  • Minor improvements for .save_table(mode="overwrite") (#298). In this release, the .save_table() method has been improved, particularly when using the overwrite mode. If no rows are supplied, the table will now be truncated, ensuring consistency with the mock backend behavior. This change has been optimized for SQL-based backends, which now perform truncation as part of the insert for the first batch. Type hints on the abstract method have been updated to match the concrete implementations. Unit tests and integration tests have been updated to cover the new functionality, and new methods have been added to test the truncation behavior in overwrite mode. These improvements enhance the consistency and efficiency of the .save_table() method when using overwrite mode across different backends.
  • Updated databrickslabs/sandbox requirement to acceptance/v0.3.0 (#305). In this release, we have updated the requirement for the databrickslabs/sandbox package to version acceptance/v0.3.0 in the downstreams.yml file. This update is necessary to use the latest version of the package, which includes several bug fixes and dependency updates. The databrickslabs/sandbox package is used in the acceptance tests, which are run as part of the CI/CD pipeline. It provides a set of tools and utilities for developing and testing code in a sandbox environment. The changelog for this version includes the addition of install instructions, more git-related libraries, and the modification of the README to include information about how to use it with the databricks labs sandbox command. Specifically, the version of the databrickslabs/sandbox package used in the acceptance job has been updated from acceptance/v0.1.4 to acceptance/v0.3.0, allowing the integration tests to be run using the latest version of the package. The ignore conditions for this PR ensure that Dependabot will resolve any conflicts that may arise and can be manually triggered with the @dependabot rebase command.

Dependency updates:

  • Bump actions/checkout from 4.2.0 to 4.2.1 (#304).
  • Updated databrickslabs/sandbox requirement to acceptance/v0.3.0 (#305).
  • Bump databrickslabs/sandbox from acceptance/v0.3.0 to 0.3.1 (#307).
  • Bump actions/checkout from 4.2.1 to 4.2.2 (#310).
  • Bump databrickslabs/sandbox from acceptance/v0.3.1 to 0.4.2 (#315).

Contributors: @dependabot[bot], @asnare, @nfx, @JCZuurmond , @larsgeorge-db

v0.12.1

26 Sep 12:15
@nfx nfx
dbfc823
Compare
Choose a tag to compare
  • Bump actions/checkout from 4.1.7 to 4.2.0 (#295). In this version 4.2.0 release of the actions/checkout library, the team has added Ref and Commit outputs, which provide the ref and commit that were checked out, respectively. The update also includes dependency updates to braces, minor-npm-dependencies, docker/build-push-action, and docker/login-action, all of which were automatically resolved by Dependabot. These updates improve compatibility and stability for users of the library. This release is a result of contributions from new team members @yasonk and @lucacome. Users can find a detailed commit history, pull requests, and release notes in the associated links. The team strongly encourages all users to upgrade to this new version to access the latest features and improvements.
  • Set catalog on SchemaDeployer to overwrite the default hive_metastore (#296). In this release, the default catalog for SchemaDeployer has been changed from hive_metastore to a user-defined catalog, allowing for more flexibility in deploying resources to different catalogs. A new dependency, databricks-labs-pytester, has been added with a version constraint of >=0.2.1, which may indicate the introduction of new testing functionality. The SchemaDeployer class has been updated to accept a catalog parameter and the tests for deploying and deleting schemas, tables, and views have been updated to reflect these changes. The test_deploys_schema, test_deploys_dataclass, and test_deploys_view tests have been updated to accept a inventory_catalog parameter, and the caplog fixture is used to capture log messages and assert that they contain the expected messages. Additionally, a new test function test_statement_execution_backend_overwrites_table has been added to the tests/integration/test_backends.py file to test the functionality of the StatementExecutionBackend class in overwriting a table in the database and retrieving the correct data. Issue #294 has been resolved, and progress has been made on issue #278, but issue #280 has been marked as technical debt and issue #287 is required for the CI to pass.

Dependency updates:

  • Bump actions/checkout from 4.1.7 to 4.2.0 (#295).

Contributors: @dependabot[bot], @JCZuurmond

v0.12.0

19 Sep 13:51
@nfx nfx
8fd3b9e
Compare
Choose a tag to compare
  • Added method to detect rows are written to the MockBackend (#292). In this commit, the MockBackend class in the 'backends.py' file has been updated with a new method, 'has_rows_written_for', which allows for differentiation between a table that has never been written to and one with zero rows. This method checks if a specific table has been written to by iterating over the table stubs in the _save_table attribute and returning True if the given full name matches any of the stub full names. Additionally, the class has been supplemented with the rows_written_for method, which takes a table name and mode as input and returns a list of rows written to that table in the given mode. Furthermore, several new test cases have been added to test the functionality of the MockBackend class, including checking if the has_rows_written_for method correctly identifies when there are no rows written, when there are zero rows written, and when rows are written after the first and second write operations. These changes improve the overall testing coverage of the project and aid in testing the functionality of the MockBackend class. The new methods are accompanied by documentation strings that explain their purpose and functionality.

Contributors: @JCZuurmond

v0.11.0

18 Sep 08:06
@nfx nfx
02ed2c8
Compare
Choose a tag to compare
  • Added filter spec implementation (#276). In this commit, a new FilterHandler class has been introduced to handle filter files with the suffix .filter.json, which can parse filter specifications in the header of the filter file and validate the filter columns and types. The commit also adds support for three types of filters: DATE_RANGE_PICKER, MULTI_SELECT, and DROPDOWN, which can be linked with multiple visualization widgets. Additionally, a FilterTile class has been added to the Tile class, which represents a filter tile in the dashboard and includes methods to validate the tile, create widgets, and generate filter encodings and queries. The DashboardMetadata class has been updated to include a new method get_datasets() to retrieve the datasets for the dashboard. These changes enhance the functionality of the dashboard by adding support for filtering data using various filter types and linking them with multiple visualization widgets, improving the customization and interactivity of the dashboard, and making it more user-friendly and efficient.
  • Bugfix: MockBackend wasn't mocking savetable properly when the mode is append (#289). This release includes a bugfix and enhancements for the MockBackend component, which is used to mock the SQLBackend. The .savetable() method failed to function as expected in append mode, writing all rows to the same table instead of accumulating them. This bug has been addressed, ensuring that rows accumulate correctly in append mode. Additionally, a new test function, test_mock_backend_save_table_overwrite(), has been added to demonstrate the corrected behavior of overwrite mode, showing that it now replaces only the existing rows for the given table while preserving other tables' contents. The type signature for .save_table() has been updated, restricting the mode parameter to accept only two string literals: "append" and "overwrite". The MockBackend behavior has been updated accordingly, and rows are now filtered to exclude any None or NULL values prior to saving. These improvements to the MockBackend functionality and test suite increase reliability when using the MockBackend as a testing backend for the system.
  • Changed filter spec to use YML instead of JSON (#290). In this release, the filter specification files have been converted from JSON to YAML format, providing a more human-readable format for the filter specifications. The schema for the filter file includes flags for column, columns, type, title, description, order, and id, with the type flag taking on values of DROPDOWN, MULTI_SELECT, or DATE_RANGE_PICKER. This change impacts the FilterHandler, is_filter method, and _from_dashboard_folder method, as well as relevant parts of the documentation. Additionally, the parsing methods have been updated to use yaml.safe_load instead of json.loads, and the is_filter method now checks for .filter.yml suffix. A new file, '00_0_date.filter.yml', has been added to the 'tests/integration/dashboards/filter_spec_basic' directory, containing a sample date filter definition. Furthermore, various tests have been added to validate filter specifications, such as checking for invalid type and both column and columns keys being present. These updates aim to enhance readability, maintainability, and ease of use for filter configuration.
  • Increase testing of generic types storage (#282). A new commit enhances the testing of generic types storage by expanding the test suite to include a list of structs, ensuring more comprehensive testing of the system. The Foo struct has been renamed to Nested for clarity, and two new structs, NestedWithDict and Nesting, have been added. The Nesting struct contains a Nested object, while NestedWithDict includes a string and an optional dictionary of strings. A new test case demonstrates appending complex types to a table by creating and saving a table with two rows, each containing a Nesting struct. The test then fetches the data and asserts the expected number of rows are returned, ensuring the proper functioning of the storage system with complex data types.
  • Minor Changes to avoid redundancy in code and follow code patterns (#279). In this release, we have made significant improvements to the dashboards.py file to make the code more concise, maintainable, and in line with the standard library's recommended usage. The export_to_zipped_csv method has undergone major changes, including the removal of the BytesIO module import and the use of StringIO for handling strings as files. The method no longer creates a separate ZIP file for the CSV files, instead using the provided export_path. Additionally, the method skips tiles that don't contain queries. We have also introduced a new method, dataclass_transform, which transforms a given dataclass into a new one with specific attributes and behavior. This method creates a new dataclass with a custom metaclass and adds a new method, to_dict(), which converts the instances of the new dataclass to dictionaries. These changes promote code reusability and reduce redundancy in the codebase, making it easier for software engineers to work with.
  • New example with bar chart in dashboards-as-code (#281). A new example of a dashboard featuring a bar chart has been added to the dashboards-as-code feature using the existing metadata overrides feature to support the new widget type, without bloating the TileMetadata structure. An integration test was added to demonstrate the creation of a bar chart, and the resulting dashboard can be seen in the attached screenshot. Additionally, a new SQL file has been added for the Product Sales dashboard, showcasing sales data for different product categories. This approach can potentially be used to support other widget types such as Bar, Pivot, Area, etc. The team is encouraged to provide feedback on this proposed solution.

Contributors: @JCZuurmond, @bishwajit-db, @ericvergnaud, @jgarciaf106, @asnare

v0.10.0

11 Sep 16:31
@nfx nfx
82826ab
Compare
Choose a tag to compare
  • Added Functionality to export any dashboards-as-code into CSV (#269). The DashboardMetadata class now includes a new method, export_to_zipped_csv, which enables exporting any dashboard as CSV files in a ZIP archive. This method accepts sql_backend and export_path as parameters and exports dashboard queries to CSV files in the specified ZIP archive by iterating through tiles and fetching dashboard queries if the tile is a query. To ensure the proper functioning of this feature, unit tests and manual testing have been conducted. A new test, test_dashboards_export_to_zipped_csv, has been added to verify the correct export of dashboard data to a CSV file.
  • Added support for generic types in SqlBackend (#272). In this release, we've added support for using rich dataclasses, including those with optional and generic types, in the SqlBackend of the StatementExecutionBackend class. The new functionality is demonstrated in the test_supports_complex_types unit test, which creates a Nested dataclass containing various complex data types, such as nested dataclasses, datetime objects, dict, list, and optional fields. This enhancement is achieved by updating the save_table method to handle the conversion of complex dataclasses to SQL statements. To facilitate type inference, we've introduced a new StructInference class that converts Python dataclasses and built-in types to their corresponding SQL Data Definition Language (DDL) representations. This addition simplifies data definition and manipulation operations while maintaining type safety and compatibility with various SQL data types.

Contributors: @jgarciaf106, @nfx

v0.9.3

04 Sep 15:41
@nfx nfx
e21699f
Compare
Choose a tag to compare
  • Added documentation for exclude flag (#265). A new exclude flag has been added to the configuration file for our lab tool, allowing users to specify a path to exclude from formatting during lab execution. This release also includes corrections to grammatical errors in the descriptions of existing flags related to catalog and database settings, such as updating seperated to "separate". Additionally, the flag descriptions for publish and open-browser have been updated for clarification: publish now clearly controls whether the dashboard is published after creation, while open-browser controls whether the dashboard is opened in a web browser. These changes are aimed at improving user experience and ease of use for our lab tool.
  • Fixed dataclass field type in _row_to_sql (#266). In this release, we have addressed an issue related to #257 by fixing the dataclass field type in the _row_to_sql method of the backends.py file. Additionally, we have made updates to the _schema_for method to use a new _field_type class method. This change resolves a rare problem where the field.type is a string instead of a type and ensures compatibility with a pull request from an external repository (databrickslabs/ucx#2526). The new _field_type method attempts to load the type from __builtins__ if it's a string and logs a warning if it fails. The _row_to_sql method now consistently uses the _field_type method to get the field type. This ensures that the library functions seamlessly and consistently, avoiding any potential issues in the future.

Contributors: @ericvergnaud

v0.9.2

03 Sep 14:26
@nfx nfx
092dbc6
Compare
Choose a tag to compare
  • Make hatch a prerequisite (#259). In this commit, Eric Vergnaud has introduced a change to make the installation of hatch version 1.9.4 a prerequisite for the project to avoid errors related to pip command recognition. The Makefile has been updated to handle the installation of hatch automatically, and the hatch env create command is now used instead of pip install hatch==1.7.0. This change ensures that the development environment is consistent and reliable by maintaining the correct version of hatch and automatically handling its installation. Additionally, the .venv/bin/python and dev targets have been updated accordingly to reflect these changes. This commit also formats all files using the make dev fmt command, which helps maintain consistent code formatting throughout the project.
  • add support for exclusions in fmt command (#263). In this release, we have added support for exclusions to the fmt command in the 'databricks/labs/lsql/cli.py' module. This feature allows users to specify a list of directories or files to exclude while formatting SQL files, which is particularly useful when verifying SQL notebooks in ucx. The fmt command now accepts a new optional parameter 'exclude', which accepts an iterable of strings that specify the relative paths to exclude. Any sql_file that is a descendant of any exclusion is skipped during formatting. The exclusions are implemented by converting the relative paths into Path objects. This change addresses the issue where single line comments are converted into inlined comments, causing misinterpretation. The added unit test is manually verified, and this pull request fixes issue #261. This feature was authored and co-authored by Eric Vergnaud.

Contributors: @ericvergnaud

v0.9.1

30 Aug 15:42
@nfx nfx
fa23371
Compare
Choose a tag to compare
  • Fixed dataclass field types (#257). This PR introduces a workaround to a Python bug affecting the dataclasses.fields() function, which sometimes returns field types as string type names instead of types. This can cause the ORM to malfunction. The workaround involves checking if the returned f.type is a string, and if so, converting it to a type by looking it up in the __builtins__ dictionary. This change is global and affects the _schema_for function in the backends.py file, which is responsible for creating a schema for a given dataclass, taking into account any necessary type conversions. This change ensures consistent and accurate type handling in the face of the Python bug, improving the reliability of our ORM.
  • Fixed missing EOL when formatting SQL files (#260). In this release, we have addressed an issue related to the inconsistent addition of end-of-line (EOL) characters in formatted SQL files. The QueryTile.format() method has been updated to ensure that an EOL character is always added, except when the input query already ends with a newline. This change enhances the reliability of the SQL formatting functionality, making the output format more predictable and improving the overall user experience. The new implementation is demonstrated in the test_query_format_preserves_eol() test case, and existing test cases have been updated to check for the presence of EOL characters, further ensuring consistent and correct formatting.
  • Fixed normalize case input in cli (#258). In this release, we have updated the fmt command in the cli.py file to allow users to specify whether they want to normalize the case of SQL files when formatting. The normalize_case parameter now defaults to the string "true" and checks if it is in the STRING_AFFIRMATIVES list to determine whether to normalize the case of SQL files. Additionally, we have introduced a new optional normalize_case parameter in the format method of the dashboards.py file in the Databricks CLI, which normalizes the identifiers in the query to lower case when set to True. We have also added support for a new normalize_case parameter in the QueryTile.format() method, which prevents the automatic normalization of string input to uppercase when set to False. This change allows for more flexibility in handling string input and ensures that the input string is preserved as-is. These updates improve the functionality and usability of the open-source library, providing more control to users over formatting and handling of string input.

Contributors: @ericvergnaud, @nfx, @JCZuurmond