Skip to content

Latest commit

 

History

History
845 lines (469 loc) · 52.4 KB

CHANGELOG.md

File metadata and controls

845 lines (469 loc) · 52.4 KB
Raw

Release Notes

0.60.0 (2024-11-10)

Features

  • add antlr grammar for test file format (#728) (752aa63)
  • add CreateMode for CTAS in WriteRel (#715) (2e13d0b)
  • update test file format to support aggregate functions (#736) (c18c0c1)

Bug Fixes

  • typo in site/docs/tutorial/sql_to_substrait.md (#735) (9cccb04)

0.59.0 (2024-11-03)

⚠ BREAKING CHANGES

  • changes the message type for Expressions field in VirtualTable

Features

Bug Fixes

  • virtualTable expression should represent a row of expression (#727) (a2df42c)

0.58.0 (2024-10-13)

Features

0.57.1 (2024-10-06)

Bug Fixes

  • add missing udt identifier in functions_geometry (#716) (f1cedd2)

0.57.0 (2024-10-02)

⚠ BREAKING CHANGES

  • This PR changes the definition of grouping sets in AggregateRel to consist of references into a list of grouping expressions instead of consisting of expressions directly.

With the previous definition, consumers had to deduplicate the expressions in the grouping sets in order to execute the query or even derive the output schema (which is problematic, as explained below). With this change, the responsibility of deduplicating expressions is now on the producer. Concretely, consumers are now expected to be simpler: The list of grouping expressions immediately provides the information needed to derive the output schema and the list of grouping sets explicitly and unambiguously provides the equality of grouping expressions. Producers now have to specify the grouping sets explicitly. If their internal representation of grouping sets consists of full grouping expressions (rather than references), then they must deduplicate these expressions according to their internal notion of expression equality in order to produce grouping sets consisting of references to these deduplicated expressions.

If the previous format is desired, it can be obtained from the new format by (1) deduplicating the grouping expressions (according to the previously applicable definition of expression equality), (2) re-establishing the duplicates using the emit clause, and (3) "dereferencing" the references in the grouping sets, i.e., by replacing each reference in the grouping sets with the expression it refers to.

The previous version was problematic because it required the consumers to deduplicate the expressions from the grouping sets. This, in turn, requires to parse and understand 100% of these expression even in cases where that understanding is otherwise optional, which is in opposition to the general philosophy of allowing for simple-minded consumers. The new version avoids that problem and, thus, allows consumers to be

Features

  • change grouping expressions in AggregateRel to references (#706) (65a7d38), closes #700
  • clarify behaviour of SetRel operations (#708) (f796521)
  • make substrait repo a go module (#712) (3dca9b5)

0.56.0 (2024-09-15)

Features

  • add optional metadata containing field names to RelCommon (#696) (5a73281)
  • define mark join (#682) (bc1b93f)

Bug Fixes

  • correct format for nullable interval_day parameters (#687) (8ae1084), closes #679

0.55.0 (2024-08-18)

Features

  • update interval_day function extensions to include precision param (#679) (28025cb)

0.54.0 (2024-08-11)

⚠ BREAKING CHANGES

  • The encoding of IntervalDay literals has changed in a strictly backwards incompatible way. However, the logical meaning across encoding is maintained using a oneof. Moving a field into a oneof makes unset/set to zero unclear with older messages but the fields are defined such that the logical meaning of the two is indistinct. If neither microseconds nor precision is set, the value can be considered a precision 6 value. If you aren't using IntervalDay type, you will not need to make any changes.
  • TypeExpression and Parameterized type protobufs (used to serialize output derivation) are updated to match the now compound nature of IntervalDay. If you use protobuf to serialize output derivation that refer to IntervalDay type, you will need to rework that logic.
  • JoinRel's type enum now has LEFT_SINGLE instead of SINGLE. Similarly there is now LEFT_ANTI and LEFT_SEMI. Other values are available in all join type enums. This affects JSON and text formats only (binary plans -- the interoperable part of Substrait -- will still be compatible before and after this change).

Features

  • add arithmetic function "power" with decimal type (#660) (9af2d66)
  • add CSV (text) file support (#646) (5d49e04)
  • add precision to IntervalDay and new IntervalCompound type (#665) (e41eff2), closes #664
  • normalize the join types (#662) (bed84ec)

0.53.0 (2024-08-04)

⚠ BREAKING CHANGES

  • PrecisionTimestamp(Tz) literal's value is now int64 instead of uint64

Features

  • add aggregate count functions with decimal return type (#670) (2aa516b)
  • add arithmetic function "sqrt" and "factorial" with decimal type (#674) (e4f5b68)
  • add arithmetic function for bitwise(AND/OR/XOR) operation with decimal arguments (#675) (a70cf72)
  • add logarithmic functions with decimal type args (#669) (d9fb1e3)
  • add precision timestamp datetime fn variants (#666) (60c93d2)
  • clarify the meaning of plans (#616) (c1553df), closes #612 #613

Bug Fixes

  • use int64 instead of uint64 for PrecisionTimestamp(Tz) literal value (#668) (da3c74e)

0.52.0 (2024-07-14)

⚠ BREAKING CHANGES

  • changes the message type for Literal PrecisionTimestamp and PrecisionTimestampTZ

The PrecisionTimestamp and PrecisionTimestampTZ literals were introduced

Bug Fixes

0.51.0 (2024-07-07)

Features

0.50.0 (2024-06-30)

⚠ BREAKING CHANGES

  • consumers must now check for multiple optimization messages within an AdvancedExtension

Features

0.49.0 (2024-05-23)

Features

Bug Fixes

0.48.0 (2024-04-25)

⚠ BREAKING CHANGES

  • min:ts has been moved to functions_datetime
  • max:ts has been moved to functions_datetime

Bug Fixes

  • duplicate declaration of min:ts and max:ts (#631) (7fc86f8)

0.47.0 (2024-04-18)

Features

  • add i64 variant for exp, ln, log10, log2 and logb functions (#628) (fef2253)
  • allow FetchRel to specify a return of ALL results (#622) (#627) (37f43b4)

Bug Fixes

  • index_in has wrong return type (#632) (4cd2089)
  • use any1 instead of T in function extensions (#629) (0bddf68)

0.46.0 (2024-04-14)

Features

Bug Fixes

  • remove implicit casts in trig extension functions (#620) (b883120)

0.45.0 (2024-03-24)

Features

  • add decimal type support for sum0 function (#610) (6bd0c7b)

0.44.0 (2024-03-03)

⚠ BREAKING CHANGES

  • Adding a NULL option to the on_domain_errors.

SQLite returns null for some inputs such as negative infinity

Features

  • add extra option for on domain errors in log functions (#536) (cbec079)
  • add ignore nulls options to concat function (#605) (55db05b)

0.43.0 (2024-02-25)

Features

  • include precision parameter in timestamp types (#594) (087f87c)

Bug Fixes

  • remove function definitions w/ invalid return types (#599) (a3b1f32)

0.42.1 (2024-01-28)

Bug Fixes

  • add missing RelCommon field to WriteRel and DdlRel (#591) (d55703a)

0.42.0 (2024-01-21)

Features

0.41.0 (2023-12-24)

⚠ BREAKING CHANGES

  • Renamed modulus to modulo.

Added options and documentation for the modulo operator as defined in math and comp sci.

Bug Fixes

  • renamed modulus to modulo; updated modulo operator defintion (#583) (aba1bc7), closes #353

0.40.0 (2023-12-17)

⚠ BREAKING CHANGES

  • The enum WriteRel::OutputMode had an option change from OUTPUT_MODE_MODIFIED_TUPLES to OUTPUT_MODE_MODIFIED_RECORDS
  • The message AggregateFunction.ReferenceRel has moved to ReferenceRel.

Features

0.39.0 (2023-11-26)

⚠ BREAKING CHANGES

    • Map keys may be repeated.
  • Map keys must not be NULL.
  • The map key type may be nullable.

This is based on the current restrictions found in the wild.

DuckDB, Velox, Spark, and Acero all reject attempts to provide NULL as a key.

Despite DuckDB specifically calling out that keys must be unique in its implementation other implementations such as Velox and Acero do not require the key to be unique so we cannot require the map key to be 1:1 with map values.

Features

  • support for simple extensions dependencies (#265) (f0ecf54)

Documentation

0.38.0 (2023-11-05)

Features

  • add least and greatest functions to functions_comparison.yml (#247) (b3071bc)

0.37.0 (2023-10-22)

Features

0.36.0 (2023-10-08)

Features

0.35.0 (2023-10-01)

⚠ BREAKING CHANGES

  • nullability of is_not_distinct_from has changed
  • The minimum precision for floating point numbers is now mandated.

Features

  • add approval guidelines for documentation updates (#553) (da4b32a)
  • add geometric data types and functions (#543) (db52bbd)
  • add geometry editor functions (#554) (727467c)
  • adding geometry accessor functions (#552) (784fa9b)
  • explicitly reference IEEE 754 and mandate precision as well as range (#449) (54e3d52), closes #447

Bug Fixes

  • specify nullability for is_not_distinct_from (#555) (30773b2)

0.34.0 (2023-09-17)

Features

0.33.0 (2023-08-27)

Features

0.32.0 (2023-08-21)

⚠ BREAKING CHANGES

  • plans referencing functions using simple names (e.g. not vs not:bool) will no longer be valid.

Features

  • add ExchangeRel as a type in Rel (#518) (89b0c62)
  • add expand rel (#368) (98380b0)
  • add options to substring for start parameter being negative (#508) (281dc0f)
  • add windowrel support in proto (#399) (bd14e0e)
  • require compound functions names in extension references (#537) (2503beb)

0.31.0 (2023-07-02)

Features

  • add a two-arg variant of substring (#513) (a6ead70)
  • add timestamp types to max/min function (#511) (6943400)

0.30.0 (2023-05-14)

⚠ BREAKING CHANGES

  • This adds an option to control indexing of components

Features

0.29.0 (2023-04-23)

⚠ BREAKING CHANGES

  • text: mark name and structure property of type extension item as required (#495)

Bug Fixes

  • referenced simple extension in tutorial (set instead of string) (#494) (b5d7ed2)
  • text: mark name and structure property of type extension item as required (#495) (7246102)

0.28.2 (2023-04-16)

Bug Fixes

  • separate strptime to fix spec violation (#493) (8c230af)

0.28.1 (2023-04-09)

Bug Fixes

0.28.0 (2023-04-02)

Features

0.27.0 (2023-03-26)

⚠ BREAKING CHANGES

  • group argument added to regexp_match_substring function

Add regexp_match_substring_all function

Resolves #466

Features

  • add regexp_match_substring_all function to yaml (#469) (b4d81fb)

Bug Fixes

  • ci: fix link to conventional commits spec (#482) (45b4e48)
  • remove duplication in simple extensions schema (#404) (b7df38d)

0.26.0 (2023-03-05)

Features

  • add script to re-namespace .proto files for internal use in public libraries (#207) (a6f24db)
  • add temporal functions (#272) (beb104b), closes #222

0.25.0 (2023-02-26)

⚠ BREAKING CHANGES

  • (add/subtract)ing an interval to a timestamp_tz now requires a time zone and returns a timestamp_tz

Bug Fixes

  • correct return of temporal add and subtract and add timezone parameter (#337) (1b184cc)
  • extension: fix typo in scalar function argument type (#445) (7d7ddf1)

0.24.0 (2023-02-12)

Features

0.23.0 (2023-01-22)

Features

  • add extended expression for expression only evaluation (#405) (d35f0ed)
  • spec: add physical plans for hashJoin and mergeJoin (#336) (431651e)

Bug Fixes

  • update extension yaml files to match type-syntax spec (#423) (0608878)

0.22.0 (2022-12-18)

Features

  • add bitwise NOT, AND, OR & XOR functions (#370) (81e34d4)

0.21.1 (2022-12-04)

Bug Fixes

  • rename regex_string_split to regexp_string_split (#393) (f9f4967)

0.21.0 (2022-11-27)

Features

  • add nested type constructor expressions (#351) (b64d30b)
  • add title to simple extensions schema (#387) (2819ecc)

0.20.0 (2022-11-20)

⚠ BREAKING CHANGES

  • optional arguments are no longer allowed to be specified as a part of FunctionArgument messages. Instead they are now specified separately as part of the function invocation.
  • optional arguments are now specified separately from required arguments in the YAML specification.

Co-authored-by: Benjamin Kietzman bengilgit@gmail.com

Co-authored-by: Benjamin Kietzman bengilgit@gmail.com

Features

  • add best effort filter to read rel and clarify that the pre-masked schema should be used (#271) (4beff87)
  • optional args are now specified separately from required args (#342) (bd29ea3)

0.19.0 (2022-11-06)

Features

Bug Fixes

  • rename version fields which conflict with sysmacros (#362) (4170bf1)

0.18.0 (2022-10-09)

Features

  • attach Substrait version number to plans (#347) (2d1bb9d)

0.17.0 (2022-10-02)

Features

  • support non-struct type class structure (#328) (dd7f9f0)

0.16.0 (2022-09-25)

Features

0.15.0 (2022-09-18)

⚠ BREAKING CHANGES

  • options were added to division and logarithmic functions.

Features

  • add options for behaviour when dividing by zero or calculating log zero (#329) (1c170c8)

Bug Fixes

  • naming: add missing arg names in functions_aggregate_*.yaml (#316) (fb92997)

0.14.0 (2022-09-11)

⚠ BREAKING CHANGES

  • option argument added to std_dev and variance aggregate functions

Features

  • add bool_and and bool_or aggregate functions (#314) (52fa523)
  • add corr and mode aggregation functions (#296) (96b13d7)
  • add median and count_distinct aggregation functions (#278) (9be62e5)
  • add population option to variance and standard deviation functions (#295) (c47fffa)
  • add quantile aggregate function (#279) (de6bc9f)
  • add string_agg aggregate function (#297) (fbe5e09)

Bug Fixes

  • mark string_agg aggregate as being sensitive to input order (#312) (683faaa)
  • naming: add missing arg names in functions_arithmetic.yaml (#315) (d433a06)
  • naming: add missing arg names in functions_datetime.yaml (#318) (b7347d1)
  • naming: add missing arg names in functions_logarithmic.yaml and functions_set.yaml (#319) (1c14d27)
  • naming: add/replace arg names in functions_boolean.yaml (#317) (809a2f4)
  • revert addition of count_distinct aggregate function (#311) (90d7c0d)

0.13.0 (2022-09-04)

⚠ BREAKING CHANGES

  • nullability behavior of is_nan, is_finite, and is_infinite has changed
  • compound name for concat has changed to concat:str and concat:vchar (one argument) to make it 1+ variadic

Features

0.12.0 (2022-08-28)

Features

0.11.0 (2022-08-21)

Features

0.10.0 (2022-08-14)

Features

  • add and_not boolean function (#276) (8af3fe0)
  • add is_finite and is_infinite (#286) (01d5428)
  • add support for DDL and INSERT/DELETE/UPDATE operations (#252) (cbb6c26)

0.9.0 (2022-07-31)

⚠ BREAKING CHANGES

  • arithmetic: Options SILENT, SATURATE, ERROR are no longer valid for use with floating point arguments to add, subtract, multiply or divide
  • function argument bindings were open to interpretation before, and were often produced incorrectly; therefore, this change semantically shifts some responsibilities from the consumers to the producers.
  • the grouping set index column now only exists if there is more than one grouping set.
  • Existing plans that are modeling cast with the cast function (as opposed to the cast expression) will no longer be valid. All producers/consumers should use the cast expression type.

Features

  • add functions for arithmetic, rounding, logarithmic, and string transformations (#245) (f7c5da5)
  • add standard deviation functions (#257) (1339534)
  • add string containment functions (#256) (d6b9b34)
  • add string trimming and padding functions (#248) (8a8f65d)
  • add trigonometry functions (#241) (d83d566)
  • add variance function (#263) (b6c3772)
  • arithmetic: add abs and sign to scalar function extensions (#244) (1b9a45f)
  • support window functions (#224) (4b2072a)

Bug Fixes

Documentation

Code Refactoring

  • arithmetic: specify FP overflow and domain options for remaining ops (#269) (de64a3c)

0.8.0 (2022-07-17)

⚠ BREAKING CHANGES

  • The signature of divide functions for multiple types now specify an enumeration prior to specifying operands.

Bug Fixes

  • add overflow behavior to integer division (#223) (cf552d7)

0.7.0 (2022-07-11)

Features

  • introduce compound (parameterizable) extension types and variations (#196) (a79eb07)

0.6.0 (2022-06-26)

Features

  • add contains, starts_with and ends_with functions definitions (#228) (a5fa851)

Bug Fixes

0.5.0 (2022-06-12)

⚠ BREAKING CHANGES

  • The substrait/ReadRel/LocalFiles/format field is deprecated. This will cause a hard break in compatibility. Newer consumers will not be able to read older files. Older consumers will not be able to read newer files. One should now express format concepts using the file_format oneof field.

Co-authored-by: Jacques Nadeau jacques@apache.org

Features

  • add aggregate function min/max support (#219) (48b6b12)
  • add Arrow and Orc file formats (#169) (43be00a)
  • support nullable and non-default variation user-defined types (#217) (5851b02)

0.4.0 (2022-06-05)

⚠ BREAKING CHANGES

  • there was an accidental inclusion of a binary not function with unspecified behavior. This function was removed. Use the unary not function to return the compliment of an input argument.

Bug Fixes

  • remove not function that expects two arguments (#182) (e06067c)

0.3.0 (2022-05-22)

Features

  • support type function arguments in protobuf (#161) (df98816)
  • define APPROX_COUNT_DISTINCT in new yaml for approximate aggregate functions (#204) (8e206b9)
  • literals for extension types (#197) (296c266)
  • support fractional seconds for interval_day literals (#199) (129e52f)

0.2.0 (2022-05-15)

Features

  • add flag FailureBehavior in Cast expression (#186) (a3d3b2f)
  • add invocation property to AggregateFunction message for specifying distinct vs all (#191) (373b33f)

0.1.2 (2022-05-01)

Bug Fixes

  • docs: use conventionalcommits to show breaking changes first (#181) (b7f2587)

0.1.1 (2022-04-28)

Bug Fixes

  • ci: cd into buf-configured proto directory (#180) (78c0781)

0.1.0 (2022-04-28)

Bug Fixes

Features