Bug fixes, de facto release candidate

This release contains several bug fixes.

The most important bug fix, perhaps, is that it fixes a long-standing regression where commands for cue cmd were allowed to be specified in non-tool files. The current implementation will require a command to be define in at least one *_tool.cue file.

Performance fixes have been put on hold for the moment to allow some stabilization before the final v0.3.0 release.

Changelog

35e19b2 cmd/cue: fix bug in -o, which would not write files for some commands
7c5d28e cmd/cue: fix race in testscript tests that use go/packages
81aa8cd cmd/export: fix help
d4cb2c8 cue: add locations for disallowed field
eb7b60d doc/ref/spec.md: fix ListLit production
fec7a9c encoding/openapi: fix title handling regression
bbe68e1 internal/core/adt: allow pattern constraints alongside fields
5049ab7 internal/core/adt: fix nil interface panic
1f9300d internal/core/adt: handle error properly for failed pattern constraint
2a937a9 internal/core/adt: improve error message
dd5083f tools/flow: add clarifications in the API comments

More bug fixes

This release has some important bug fixes, fixing some regressions in disjunctions and trim, mostly, among some other bug fixes.

Some of the regressions in this release slipped in because a swat of tests was accidentally disabled in the move to the new evaluator. All former tests are now reenabled and a few more corresponding issues have been resolved.

There are some performance issues remaining. We are considering leaving those in and do a v0.3.0 release, addressing these in a subsequent release. In this case, we are getting fairly close to a v0.3.0.

There will still be a beta.7 release, which addresses a regression, allowing commands in non-tool files, that we would like to address in a separate release.

Changelog

91abe0d ci: fix goreleaser config
a113048 ci: move to using go1.16
f014c14 ci: set a "trybot" tag for trybot repository dispatch flow
5d3afa9 cmd/cue/cmd: correct documentation for -d flag
de1f80c cmd/cue/cmd: remove skip tests
304a3e0 cmd/cue/cmd: use unify semantics merging across packages
ae43812 cmd/cue: make get go operate on a best-efforts basis
cf5c48e cmd/cue: only ignore type and parse errors with get go
7326cfa cue: Value.Equal should not only consider values
830cfbc cue: remove build.Context reference
17ea1d5 doc/ref/spec.md: a few fixes
f5a19ef doc/ref/spec.md: fix escaping
3ca8680 doc/ref/spec.md: fix int_lit 0
e4d0d13 doc/ref/spec.md: remove faulty use of \x in double-quoted string
7df9fa4 doc/tutorial/basics: fix example
b51368e doc/tutorial/kubernetes: add required go mod init step
430c3dd encoding/jsonschema: fix dropping of struct for additionalProperties: true
1347dd4 general: add .unity-bin to gitignore
a0e1970 general: add unity configuration to codereview.cfg
351a77e internal/core/adt: add more error positions
79644c3 internal/core/adt: allow hidden fields alongside embedded lists
70e2f2e internal/core/adt: discard errors in field verification
452c7bd internal/core/adt: don't count optional fields as proof of struct
3d926af internal/core/adt: don't eliminate incomplete disjuncts prematurely
1589a07 internal/core/adt: get default when comparing ot bottom
e12e26e internal/core/compile: fix let resolution bug
77e6b51 internal/core/compile: reenable disabled tests
2c86835 internal/core/eval: minor performance improvements for disjunctions
e1e7031 internal/core/runtime: decouple cue API from Runtime
c6b7ab2 internal/cuetest: add support for issue-based conditions/checkers
38f0f63 pkg/internal: fix decimal and string list errors
af3c9dc pkg/list: fix list.Unique dedupping
dcf932f tools/trim: fix too aggressive elimination for disjunctions

v0.3.0-beta.5

This release has some important bug fixes. It also is a bit stricter in considering some errors that were previously evaluation-time errors to be compile time errors.

There is also a slight change in default handling, which should not affect most configurations.

New compile-time errors

The expression >10 <20 is syntactically valid CUE, but can never meaningfully evaluate to something. It parses as >(10 < 20) and thus evaluates to >true, which is undefined. The error messages associated with this were often confusing to the user. Also, leaving this to be detected during runtime would sometimes hide the error away.

Consistent with the “fail early” mantra, these cases are now detected and reported by the compiler.

Backwards compatibility

Defaults

Previously, for

#def: {
   *{} | {a: int} | {b: int}
   *{} | {c: int} | {d: int}
}
x: #def & {a: int}

would officially resolve to

x: {a: int} | {a: int, c: int} | {a: int, d: int}

Thanks to a hack in the current v0.3 implementation, the first disjunct was made the default and all resolved fine. We have now slightly changed the definition of defaults to handle this case.
Basically, now if during a unification all terms of a disjunction that are marked as default are eliminated, the remaining terms will behave as if they originated from an unmarked disjunction.
In practice this means defaults behave as users typically expect them to behave. This may also cover cases that were not previously handed

Note that v0.2 solved things differently: if in the result of a disjunction one term subsumes another, then the latter could be dropped. So in v0.2, the answer would be x: {a: int}, even in the absence of defaults. The problem with this approach is that it could lead to confusing answers and that the default could not always be accurately determined. A goal of v0.3 was to have a simpler story for disambiguation and let simplifications be merely a matter of optimization.

There is still a slight discrepancy in the current implementation and the spec that may cause unexpected results. We expect this to be sufficiently rare and we intend to address this in a rework of disjunction computation, aimed at further performance optimizations.

Changelog

1b03c9f ci: move to non-Docker version of goreleaser
ae51469 ci: skip known flakey test on go1.15 windows
500e431 doc/ref/spec.md: add missing binary_digits
89cada6 doc/tutorial: address cuelang/cuelang.org#126
77b475f doc: fix install instructions in main README and contributing
25ba1d9 internal/core/adt: comment out leak test
b25147c internal/core/adt: control fixes for if clauses
267379b internal/core/adt: fix trim bug
48d255a internal/core/adt: some optimizations for pattern constraints
87d43b7 internal/core/adt: strip "notDefault" status for disjunctions with unused defaults
760aa11 internal/core/compile: make >10 <20 a compile error
b20ac5a internal/core/convert: fix conversion from float
bcd752a internal/core/eval: handle embedded top correctly
a5daa16 internal/core/export: fix quoting of definitions
cb4fae9 internal/cuetest: consolidate testscript/cuetxtar helpers
bf6c3fc internal/cuetest: move Kubernetes related helper to where they are used
15934d9 tools/trim: add tracer
1d9aa17 tools/trim: don't trim shared nodes
af5ea97 tools/trim: fix defaults bug

Performance improvements and bug fixes

This release has some important bug fixes related to performance, disjunctions, and performance of disjunctions.

Backwards compatibility

As a result of the disjunction rework, we introduced another internal workaround for dealing with the fact that there is no good way to represent proto oneOf fields in CUE. Right now this is done as

*{} | { a: int } | { b: int }

or

*{} | { a?: int } | { b?: int }

The need for the default is because one cannot say that a field must exist. So the default is used to disambiguate. This works well for one set of oneOfs, but not in the presence of multiple oneOfs.

A previous hack would unmark a disjunction as having a default if all but one of the defaults remains. After the critical performance improvements, this still worked, but only because these changes introduced a bug. Now this is fixed we can no longer rely on it. To keep protobuf working, there is now another hack that recognizes this pattern and adjusts defaults in a similar manner.

This is an unacceptable solution, of course. The idea is to allow specifying required fields instead, somewhat like this

{} | { a!: int } | { b!: int }

and transition to this model.

The required field specification has many benefits and solves several modeling limitations of CUE we’ve encountered since its inception. It can also almost, if not entirely, eliminate the need for ?, provides a solution for some limitations of the proposed query syntax, and generally provides better alternatives for cases where one finds the need to abuse defaults.

We plan to officially introduce this construct in a proposal soon.

Changelog

09cd07b all: add tests for the GitHub workflow process
f0d1fa4 ci: add first version of copybara workflow
10a48e9 ci: drop go1.13 from build matrix; add go1.16rc1
e097927 ci: fix broken build
34333ce ci: fix job dependency in test workflow
726d93e ci: fix start message of CI jobs for CL builds
9b7fc4b ci: fix workflow definitions
3088513 ci: only go test -race on latest Go+linux, and master
5b3d551 ci: refactor CI setup
2338b59 ci: rename repository dispatch workflow
ef59f61 ci: update version of Go for go generate to 1.14.14
257486c cmd/cue/cmd: allow fmt to pass with errors
dfce25c cmd/cue: add goproxytest instance for script tests
b8d9972 cmd/cue: allow CUE_UPDATE to update cmd/cue/cmd testscript tests
a616925 cmd/cue: embed packages required for get go checking
22aafc2 cmd/cue: fix typo in cmd documentation headline
37c16cc cmd/cue: use gotooltest within testscript tests
d9d0487 cue/scanner: fix JSON conformance
ba8eb37 cue: allow LookupDef on embedded scalars
c6f1287 cue: deprecate Merge
f73e78f doc/../43_schema.txt: remove unnecessary word
6c3be7f doc: fix typo in types tutorial (closed -> close)
2df92cb doc: remove duplicate command from Kubernetes tutorial
b360fe4 encoding/gocode: use FieldByName in generated code
7110020 general: move from godoc.org to pkg.go.dev for documentation
d5d5fd1 general: remove junk files
63104c8 internal/core/adt: "fix" disjunction resolution
2ca6c5e internal/core/adt: cache evaluation results for dynamic fields
c866a51 internal/core/adt: don't check cycles for inlined arcs
6b96001 internal/core/adt: fix benign memory leak
b8b4892 internal/core/adt: fix bottom cycle error
c091274 internal/core/adt: fix bug that left errors of incompleted expressions unreported
9c5489f internal/core/adt: fix memory leak
ee78ec3 internal/core/adt: fix nil panic
8d537aa internal/core/adt: fix regressions in disjunct elimination
e6a2fb0 internal/core/adt: fix special cycle condition
bedb047 internal/core/adt: more fixes to cycle handling
8334a9e internal/core/adt: more merging of adt and former eval
d0124e2 internal/core/adt: refactor node lookup
0ad02fd internal/core/adt: restrict notification
d2aa995 internal/core/export: don't embed proper conjuncts
76a72c4 internal/core/export: fix optional output bug
fdccfbc internal/core/export: retain top in embedded scalars
34587b9 internal/core/export: support package and struct attributes
dfaea59 internal/filetypes: fix test breakage
10180b6 pkg: consistently use 0666 as perm when creating regular files
f55f17e tools/fix: eliminate x & _ and unnecessary parens
611d622 tools/flow: replace package doc example with real example

v0.3.0-beta.3

This release addresses some important bug fixes, including a performance regression. It also sports a complete reimplementation of trim, utilizing the possibilities of the data structures of the new evaluator.

cue trim

The biggest new feature of trim is that definitions now can remove fields of non-definitions. Other than that, it mostly is just bug fixes. The data structures of the new evaluator make it considerably easier to implement, leaving less than a quarter of the code.

encoding/protobuf

This package now links in the full set of builtins, obviating the need for API users to have to do this.

Changelog

fdc7010 cmd/cue/cmd: fix cue cmd crash
a8de61c cue/testdata/cycle: fix another cycle bug
28cfa74 cue: exempt hidden fields also in Unify and UnifyAccept
ca5d2db doc/tutorial/kubernetes: update files of Kubernetes demo
f0c025c encoding/protobuf: enforce loading builting packages
08e814f internal/core/adt: add support info for trim
8e6e795 internal/core/adt: change closeInfo.embedded to a flag field
0e401d6 internal/core/adt: fix performance issue with disjunctions
82bda84 internal/core/adt: treat files as belonging to same struct
c58e0ff internal/core/eval: strip arcs for nodes with structural cycles
0f570a9 internal/core/validate: pick up non-concrete values in defaults
f03928d tools/fix: allow fixing non-essential simplification
6898897 tools/trim: implement using adt

v0.3.0-beta.2

This release predominantly addresses some regressions and bug fixes. There are also some additional small performance fixes.

Backwards compatibility

There are a couple of bug fixes that may lead to backwards incompatibilities.

Disjunction disambiguation and “incomplete” errors

There were some bug fixes where disjunctions were discarded as duplicates prematurely. This was already an issue since v0.3.0-alpha1. As a result, you may now get an “incomplete value” error where before you didn’t. For instance, {a: int, c?: int} | {a: int, b?: int} now will remain as two separate disjuncts, whereas previously it was assumed evaluation was done too early that the optional fields were irrelevant.

In v0.2.2 this would also not be disambiguated. There are, however, still some differences between v0.2 and v0.3. Pre-v0.3 CUE used subsumption to disambiguate disjuncts. Subsumption is inherently imprecise and allows false negatives. This would inherently make it hard to predict when values were disambiguated. As subsumption was used at no other place during evaluation, v0.3 takes a simpler approach to only check for equality, whereby optional fields are only considered equal if they originate from the same original structs.

The general idea is that one should typically not have disjunctions of structs without those having discriminator fields (i.e. the disjuncts would fail if unified with each other, for instance with a kind field with a specific value, like K8s does). If one still needs this, then one of the structs should be marked as default to disambiguate.

In other words, v0.3 may give an “incomplete error” where v0.2 did not. For instance: #Fields: _ | [string]: #Fields would normalize away with v0.2, but won’t with v0.3. The simple solution is to use _ or {...}. The v0.3.0-alpha releases had a bug that would prevent these alternatives from working properly, but that has been fixed as of beta.1.

We may still consider allowing more disambiguation at the point of making values concrete. This will need to be considered carefully, though. We may also allow some compile-time optimizations, like translating _ | X to _. We have not done so yet, as a set of rules for such translations has the danger of making disambiguation arbitrary; we would like If people have trouble migrating to v0.3.0-beta.2, we can prioritize such simplification rules for beta.3.

We intend to come up with a set of clear and predictable rules before v0.3.0 on when CUE may disambiguate disjunctions. Until that time, please ensure that disjunctions with structs are disambiguated using defaults and discriminator fields. This should generally be considered to be proper style and highly recommendable anyway. Note that any protocol buffer and all K8s structs, for instance, follow a pattern that can be disambiguated easily.

Stricter closedness as a result of a bug fix

The value #B: {#A} & {a: 1} is now interpreted as #B: #A & {a: 1}. Previously, this was interpreted as #B: {#A, a: 1}, allowing field a regardless of whether #A allows it or not. This was a leftover of the old closedness rules of pre-v0.3.0. The old rules were somewhat convoluted and based on user feedback and experience we simplified the rules and made them more intuitive for v0.3.0.

API

We have added the list.Concat and list.Repeat builtins in anticipation of removing the * and + operators for lists in CUE. These operators are quite unintuitive as it is not clear how they interact with closedness, which has raised repeated questions. They also complicate the implementation. This has been mitigated by, internally, translating list addition to list of two list comprehensions. They also complicated automated analysis, which is harder to work around. Better to get rid of them altogether.

The next steps for the transition is to let cue fix automatically rewrite list addition and multiplications in terms of their builtins. Ultimately list addition could be replaced with a query shorthand, like [a] + [b] -> [ a.*, b.* ], using a.* as the hypothetical notation for all values of a.

Performance

We have seen some considerable performance improvements for some configurations. There is still plenty of room for improvement though, both algorithmically, which can bring long-running evaluations down to below a second or two, and by reducing the constant overhead, by itself allowing a running time reduction of at least another 90% in many cases. So if there are any pressing performance issues, we would love to hear about them.

Changelog

ed7d254 ci: delete CL build branches once workflow is complete
8c41622 ci: fix dispatch workflow
ccbfbd2 ci: move to build branch model
237a4ae ci: use the latest GitHub workflow definition
f1b1804 cmd/cue/cmd/getgo: don't be clever
ff48658 doc/ref/spec.md: disallow new definitions in closed structs
172f006 doc/ref/spec.md: get rid of list operators
5691606 internal/core/adt: add test for incomplete references in lets
bef7b26 internal/core/adt: fix cycle bug
a086f74 internal/core/adt: fix disjunct dedupping regression
ae1203c internal/core/adt: fix overzealous disambiguation
826bd7b internal/core/adt: fix regression in close builtin
34a91b9 internal/core/adt: fix some regressions
663939e internal/core/adt: improve errors for validator conflicts
13b68c8 internal/core/adt: minor cleanup
355cccd internal/core/adt: precompute optional field type info
9eee235 internal/core/adt: remove special "tentative" mode
2ef72d8 internal/core/eval: fix result of closed embedding
03aab97 internal/core: re-allow new definitions in closed structs
0ffde15 pkg/internal: fix error codes for decimal and string lists
867f71f pkg/list: add builtins for list concatenation and repetition

Performance improvements

This release is focussed predominantly on performance. It includes several algorithmic improvements that eliminate possible exponential processing and in general takes more care to avoid recomputation.

Aside from algorithmic changes, these changes also prepare for numerous further optimizations. The algorithmic changes already already give enough of an improvement (especially for some large files) to warrant another release.

Language change

This release also introduces a slight language change regarding disjunctions. Consider

x: *1 | a
a: *2 | 3

Previously, the result of x would be *1 | *2 | 3, in other words, a default value of 1 | 2. With the new semantics, the default in this case will be 1. In general, with the new semantics, if a disjunction has a term with a default, the other terms can never result in a default value.

Superficially, this makes it easier to see where defaults may come from and eliminates the need to check all terms of a disjunction. But this change also prevents some unintuitive behavior and even cases where making values more specific may lead to a less specific result.

Backwards compatibility

The changed disjunction rule may cause backwards incompatibility.

There are some minor bug fixes as a result of these changes that may change the behavior For instance, [...] | [] was previously normalized to [].

There are also some changes to error messages as a result of these changes.

Changelog

cc41853 doc/ref/spec.md: change disjunction rule
d5b4ca8 internal/core/adt: add logging utilities for debugging
0c3791e internal/core/adt: add memory management
008d37d internal/core/adt: catch errors for circular failures
b619e0c internal/core/adt: consider closed status for equality
396202c internal/core/adt: don't pre-expand disjunctions
69e0c96 internal/core/adt: introduce HasDefaults for Disjunction
514082a internal/core/adt: keep nodeContext in Vertex
a595db2 internal/core/adt: skip partial unify if unnencessary
02ff29b internal/core/convert: fix faulty label update
778719e internal/core/eval: breadth-first disjunction resolution
2be20f8 internal/core/eval: do not precompute matchers
399ba76 internal/core/eval: export fieldSet fields
0ef6caa internal/core/eval: make arcMap a list
1fa69d5 internal/core/eval: recursive disjunctions
9ca9e04 internal/core/eval: separate out dynamic fields
0e1e7b5 internal/core/eval: small cleanup of evaluation code
062b3d8 internal/core/eval: sort empty disjunction message to top of group
359685a internal/core/eval: use env from fieldSet
d4c600f internal/core/export: extract docs from nested Vertex values
c62d8a0 internal/core: merge adt with eval package
cea55b2 internal/core: reimplementation of closedness algorithm
93c79a3 tools/flow: record dependencies per value based on path

Closing the gap with the language spec

This release is characterized by closing some significant remaining gaps in with the language specification.

There are the usual bug fixes and improvement to error messages, though.

Language

The language has now been brought closer to the spec, implementing some of the remaining gaps. There have also been some simplifications that will allow further generalizing the language down the road more easily and without introducing backwards incompatibility.

Multiple comprehensions per list

As the spec already allowed, it is now possible to include multiple comprehensions per list.

For instance:

src:    [ "John", "Mary" ]
cond: true
a: [
    for x in src { x },

    if cond { "Sasha" }
]

The list operators (concatenation and multiplication) are now also implemented in terms of comprehensions. There is a great deal of confusion about list operators (is the result open, closed, when do overlapping constraints apply?). This confusion is eliminated by writing the equivalent as list of multiple comprehensions. Our intention is to remove these operators and have comprehensions (and later queries) be the only way in which lists can be concatenated or multiplied. Of course, cue fmt will help in the transition there as usual.

Embedded scalars

Just as it is possible to write

[ #a, #b ]

#a: 1
#b: 2

at the top of a CUE file, it is now also possible to have an embedded scalar along with definitions in any struct:

x: {
    [ #a, #b ]

    #a: 1
    #b: 2
}

Originally, the top-level construct was allowed to make CUE strictly compatible with JSON. Generalizing this to any struct makes the use of imported file objects consistent with normal structs. Moreover, it turns out that this construct can be quite useful overall.

Old CUE code is not affected by this change.

Semantics of [pattern]: value fields (no change after all)

The spec recently included a change that proposed to use the JSON schema semantics for pattern constraints ([k]: v), rather than the old CUE semantics. To aid in the transition, cue fmt rewrote these fields to be enclosed in curly braces, which mimics the old semantics under these planned rules.

It has now been decided to keep the old semantics and stop rewriting old uses.

As it turns out, the old semantics was used quite a bit and was actually the majority use case. The argument previously was that the JSON schema semantics was not easily expressible in terms of the old semantics, but that the old semantics is easy to simulate with the JSON schema semantics.

Instead we now intend to allow the JSON schema semantics additionally by writing [...k]: v. The nice thing here is that any pattern which is to be matched additionally, is now always preceded by a .... It also is a nicer generalization of the syntax.

But above all, it removes the need for a potentially painful transition.

Removed integer division operators

CUE allowed integer division and remainder per the div, mod, quo, and rem operators. These were relatively rarely used and complicated the CUE syntax, formatting and other parts of various subsystems.

Moreover, they made it lexically impossible to allow indexing of the form a.3 (instead of a[3]) down the road. We are not certain if we will allow this, but allowing it makes the syntax considerably more regular and consistent. With this change, we have the option to do so in a backwards compatible way.

These operators have now been replaced with builtins. As usual cue fmt will rewrite old uses to use the builtin for some time to come. It will use the __foo variant to avoid shadowing issues. These can be manually rewritten to remove the leading underscores if necessary.

Variable argument builtins

The model for builtins, which are still underspecified in the spec, is narrowed down more. It is now possible to define variable argument builtins.

The main reasons:

1. Variable argument builtins are quite useful in being able to extend APIs in a backwards compatible way,
2. It allows introducing some environment dependence in a clean, intuitive and nonetheless hermetic way.

The first target was pkg/path which now supports multiple OSes (see below).

Bug fixes

Errors in nested comprehension clauses were sometimes dropped, causing a failure in a nested if clause to be silently ignored. This has now been fixed.

Other fixes includes crashes related to let clauses and bytes interpolation.

Error messages

There are various areas where error messages have been improved.

cue fmt

There are various printing bugs that were fixed. Overall the stability of fmt should have been improved notably.

API

The API got a few fixes related to backwards compatibility and should now be more like v0.2.0 where this makes sense.

pkg/path

This package has now been extended to support multiple operating systems simultaneously. Most builtins have been extended to allow an additional optional argument to indicate the OS for which the builtin is to be applied. For instance:

path.Split("foo\bar", os)

will now return a different value depending on whether os is path.Windows or path.Unix. The os argument can also be omitted, in which case it defaults the existing behavior, which is identical to path.Unix.

Note that os will not default to the currently running OS. This would make the behavior non-hermetic. The idea is that the existing injection mechanism can be used (or a small extension of it to make this specific case easier) to inject an environment-specific value.

Backwards compatibility

The cue tool is now a bit more aggressive to panic if it encounters an internal inconsistency error. In return, it now allows the user to define a CUE_DEBUG=0 environment variable to disable this. In many cases the crashes can be ignored while still getting useful results.

Not using this option and filing bugs can help improve the stability of CUE though.

Main backwards incompatibility to note:

• removed integer division operators (now builtins). This can be fixed by running cue fmt.
• some of the bug fixes may cause CUE files that previously succeeded erroneously to fail now.

Changelog

e77ccb1 cmd/cue/cmd: don't bail for non-matching files
fecdd83 cmd/cue/cmd: ignore possible top-level tasks
bbe493a cmd/cue/cmd: more position information for field errors
fe73e8f cue/format: format let clause
110d0bf cue/format: pad floats with leading or trailing periods
fac5d2e cue/load: allow ./... to load all packages
555fb73 cue/parser: fix comment placement
1c904cc cue: allow multiple comprehensions in lists
8872b98 cue: fix PathCorrection tests
6c49cf0 cue: implement embedded scalars
d5177fd cue: map API values to more native ADT
b687b7f cue: remove aliases used for refactoring
19f6f3f doc/ref/spec.md: perpare for allowing a.4.f
f52a0ed doc/ref/spec.md: revert to the old semantics of [k]: v
76ea22c internal/core/adt: add Assert for debugging
37293f9 internal/core/adt: add methods for concreteness
41afc87 internal/core/adt: automatic renaming
2d568d3 internal/core/adt: better message for permanently non-concrete values
f62bfed internal/core/adt: clean up Builtin and Validator semantics
4db8ffb internal/core/adt: do list artihmetic with comprehensions
cbcb701 internal/core/adt: error message improvements
d0dd888 internal/core/adt: improve error message for optional fields
08a1652 internal/core/adt: introduce base value type
6de877a internal/core/adt: make progression check an assert
8b13752 internal/core/adt: move decimal logic into one place
b8c852d internal/core/adt: pass errors up in nested comprehensions
36f2051 internal/core/compile: add integer division builtins
9c6ded8 internal/core/compile: don't panic for incorrect labels
13e4af1 internal/core/compile: fix issue in and builtin
ff3e32f internal/core/convert: allow results of math to be integer
31896af internal/core/eval: add more tests for embedding
d49777f internal/core/eval: better incomplete error messages
55602d9 internal/core/eval: fix closedness issue for lists
9f1fec6 internal/core/eval: hoist vertex processing logic
cefe38b internal/core/eval: keep semantics for [K]: T
acc35f1 internal/core/eval: track fieldSets for data values
e7e27e0 internal/core/export: allow showing recursive errors.
039fb59 internal/core/export: fix bytes printing bug
2ea30a2 internal/core/export: fix crash with let comprehension export
16809ea internal/core/export: support byte interpolations
63594ef internal/core/runtime: assign first label to _.
0811662 internal/core/subsume: tests embeded scalars
a4d0928 internal/core/subsume: various bug fixes
136e51b internal/core: allow variable arg functions
e841714 internal/core: move equality checks from eval to adt
d174bc0 internal/cue/adt: remove use of Default in node
fa6308f internal/encoding: don't clobber files on crash
ec6f95d internal/filetpes: improve error messages
fb3b02e pkg/internal/builtin: use normal formatting for builtin test results
a50960c pkg/path/testdata: adapt Go implementation
a2692ef pkg/path/testdata: move files to minimize diffs
c77b9b0 pkg/path/testdata: stage filepath files from Go
85f8b46 pkg/path: activate OS-dependent version
64434c4 tools/fix: rewrite integer division
d5ced74 tools/flow: ignore faulty tasks outside the root

Bug fixes and error message improvements

This release is mainly focused on various bug fixes and error message improvements. This change also continues to factor out code from cmd/cue to make the functionality available outside of this tool.

Below is a selection of the fixes.

Although this is officially alpha, it is probably a more solid release to use than v0.2.2. For this reason the "Pre-release" tag has been left off from this release.

Language

Cycle handling

Various bugs related to cycle handing were fixed, including hangs and spurious detection of cycles.

Closing gaps with the specs

Hidden fields are now scoped per package.

Package pkg/sort

Sort now always uses stable so that reproducible outcomes between different compiles of CUE can be guaranteed.

API

cue.Path

The introduction of #-style definitions are not handled well by the old API. This is a consequence of them living in a separate namespace, which is not expressible in the old API.

The new cue.Path type is introduced to start addressing this. The first method that supports this is LookupPath.

This API is designed to be compatible with the planned query extension.

Package tools/flow

This new package allows arbitrary user-defined CUE schema to be interpreted as task definitions, automatically handling dependency analysis and scheduling tasks based on these dependencies.

This was written from scratch and makes use of a new native dependency analysis package, allowing for more precise analysis.

cue command

Go bridge

The translation of CUE to Go is now brought in line with Go’s encoding/json, fixing some bugs that caused the interpretation to differ.

cue cmd

This now uses the new tools/flow package, fixing a bunch of outstanding bugs. Most notably, it now allows dynamic task creation, for instance by using comprehension, and depending on such dynamic tasks: $after should now only be needed if there are really no dependencies at the CUE level.

Changelog

b4aa96d all: implement hidden identifiers scoped per-package
aa87887 ci: drop use of set-env in CI scripts
4cda4a3 cmd/cue/cmd: allow inlined embedding of pointer types
c715b94 cmd/cue/cmd: use tools/flow
e05eee7 cue/errors: correct handling of wrapping list errors
54b13db cue/literal: fix multiline quotes
6165736 cue: allow access to Selectors of Path
f4f2a02 cue: dig up wrapped error in valueError
74329a1 cue: doc improvements
56eb4b1 cue: generate error node rather than panic on faulty output.
03abe87 cue: remove most uses of xerrors
409dacf cue: support getting and looking up paths
fcd96de doc/ref/spec.md: define "regular"; fix typos
8661255 encoding/protobuf: fix trailing comment in oneof bug
b5821dc interanal/core/runtime: move building logic from cue package
b886b0f internal/core/compile: fix mutual dependent let clauses
3585705 internal/core/convert: fix for embedded structs are not honored
b99fd08 internal/core/cue: fix hang on cycles in embeddings in comprehensions
c60e115 internal/core/dep: first stab at precise dependency analyzer
7463d11 internal/core/dep: support dynamic dependencies
1888d65 internal/core/eval: fix bug in cycle handling
1d8c688 internal/core/eval: fix bug with nested embeded comprehensions
aee9955 internal/core/eval: fix cycle bug for comprehensions
1a2105e internal/core/eval: fix hang in cyclic in comprehension evaluation
3cfb4ab internal/core/eval: fix irregular dynamic fields
d1cbe10 internal/core/eval: fix spurious cycle detection
635fbdd internal/core/eval: fix spurious cycle for bulk constraints
7980ec5 internal/core/eval: handle disjunction failure more properly
ab0a2f3 internal/core/eval: re-allow ... with embeddings
30704a7 internal/core/eval: reduce per-node allocations
1e0faf0 internal/core/export: dedup let clauses for merged structs
187c734 internal/core/export: handle alias and lets
cd621ff internal/cue/eval: fix faulty status in tentative result
dbfa73b internal/cuetxtar: allow multiple golden files
f2a4a42 internal/filetypes: hard-wire common case
737a103 internal: hoist a single embedding in ToExpr
99d18dc pkg/list/sort: always use stable sort
00f345b tools/flow: API for CUE-based workflow engine
d3ff4a1 tools/flow: add IgnoreConcrete option

Language refinements and fixes

This is a relatively small release. But given that it contains some significant bug fixes and performance improvements, it seemed useful to get these out sooner.

This also

Closedness refinements

The rules surrounding closedness have been further clarified in the spec where this was previously ambiguous. Most notably, it was previously unclear whether B.b in

#A: {a: int}

B: {
    #A
    b: c: int
}

should be closed or not. This is now clarified to it not being closed. In other words, fields added alongside an embedded definition that were not part of the original embedding are not closed automatically.

This does not change the current behaviour.

There was also a bug fix related to closing structs that could leave struct open if they were referenced through a selector path like #A.b, where the reference to #A failed to trigger a close.

There remains one known gap in the implementations compared to the spec: embedding a closed struct will currently cause the field to be closed, whereas the spec says that the struct in which it is included should already be closed. This will be a backwards-incompatible change.

field not allowed error

As far as is known, aside from the spec deviation mentioned above (which is somewhat esoteric), there are no more known deviations that result in a spurious error of this kind.

There is, however, one known case where this error seems to appear erroneously. Consider the following:

#a: {b: int} | {}
b: #a & {b: string}

This may still result in a “field b not allowed” error. The problem is that the error reporting around disjunctions is still lacking. What happens is that only the error of one of the failed disjuncts is reported. In this case it fails on the first disjunct (as int != string) and the subsequently fails on the second, which doesn’t allow b. As only one of the errors is reported, it may appear that it fails on the first disjunct, which is just confusing.

The intention is to considerably improve disjunction errors soon. In the meantime, the added position information for conflict errors will hopefully help a hand to disambiguate such errors.

let clauses are now allowed in comprehensions

This has been a part of the spec for a long time, but now has finally been implemented.

... anywhere in struct

This was already allowed by the spec, compiler, and evaluator, but now is also allowed by the parser.

Simple string literals now allowed as selector

Another thing that has been allowed by the spec for a while is the use of string literals as selectors. So a.”foo”.bar is now allowed. Only single-line strings without interpolations are allowed at the moment (as per the spec).

Command line tool

cli.Ask

This is now supported.

Error reporting

Error reporting has been improved further. Most notably, positions for conflict errors (as resulting from 3 & 4), now have positions added and should be on par with v0.2.2. This also adds the fundamentals for adding richer position notation.

Performance

This release fixes one big performance issue, which could lead to exponential runtime for disjunctions. This can have a big impact for many configurations. For one user, it reduced the running time from 5 hours to less than 2 seconds.

As part of this fix, we now introduced a benchmark suite.

There are tons of significant performance improvements still possible, so this is not the last of it.

Notable Bug fixes

String handling

CUE used to piggyback on Gos strconv.Quote` for quoting strings. This was not ideal, as CUE and Go strings are not the same. It also resulted in some bugs. This is now fixed. This results in better string formatting overall.

Handling of files named -

The cue command line tool sometimes hung when a directory contained a file named -. This is now handled.

Yaml version

Until now, CUE would output YAML in version 1.2 and read it in version 1.1. And it seems that the “right thing to do” is to implement version 1.2 these days. This has now provisionally been fixed to read it mostly as version 1.2. A more proper implementation would be to replace internal/third_party/yaml to use the yaml.v3 package.

This means, for instance, that the value no is now treated as a string instead of the boolean value false. (And thus the demo on cuelang.org will have to be replaced.)

Message-level options in Protobuf

Handling of message options in .proto files would previously fail. They are now translated into declaration options in CUE with the format: @protobuf(option name=value).

Backwards compatibility

YAML

As YAML 1.2 is not backwards compatible with YAML 1.1, inputs to CUE that relied on a YAML 1.1 interpretation may now yield different results.

Closedness rules

The refinement of the rules of closedness are largely backwards compatible (and what is not is not yet implemented), but some of the bug fixes may give slightly different results nonetheless.

Changelog

8ad07fa Readme.md: remove Go Report Card
2ac4d85 cue/format: indent multiline string literals based on context.
c886094 cue/literal: implement CUE-specific quoting
11b507c cue/load: don't hang on files named -
6354c40 cue/parser: allow ... anywhere in struct
241c5bf cue/parser: allow let clause in comprehension
80f0bee cue/parser: better error message for old-style definitions
732b6d5 cue/parser: support string selector labels
e9286f0 doc/ref/spec.md: fix builtin func
ff306b7 doc/ref/spec.md: fix spec bug and broken link
15e8a05 doc/ref/spec.md: refinement of closedness
65468d5 doc/tutorial/kubernetes: update import
3af2683 encoding/protobuf: parse message options
d23402c internal/core/compile: fix alias resolution bug
742593f internal/core/compile: fix hidden definition bug
da69dc0 internal/core/eval: ... does not apply to defintions
0eb7cc5 internal/core/eval: add more positions to conflict errors
3914ef8 internal/core/eval: check for nil pointer
03fa5a8 internal/core/eval: fully evaluate nested unification
f0df4df internal/core/eval: performance: unify an arc only once per node
1fdc02a internal/core/eval: simplify validators upon evaluator
1c29042 internal/third_party/yaml: drop support for non-standard scalars
5bfe37d pkg/tool/cli: support ask task