Skip to content
This repository has been archived by the owner on Nov 18, 2021. It is now read-only.

Closing the gap with the language spec
Compare
Choose a tag to compare
@mpvl mpvl released this 16 Dec 15:26
· 328 commits to master since this release
v0.3.0-alpha6
cefe38b

This release is characterized by closing some significant remaining gaps in with the language specification.

There are the usual bug fixes and improvement to error messages, though.

Language

The language has now been brought closer to the spec, implementing some of the remaining gaps. There have also been some simplifications that will allow further generalizing the language down the road more easily and without introducing backwards incompatibility.

Multiple comprehensions per list

As the spec already allowed, it is now possible to include multiple comprehensions per list.

For instance:

src:    [ "John", "Mary" ]
cond: true
a: [
    for x in src { x },

    if cond { "Sasha" }
]

The list operators (concatenation and multiplication) are now also implemented in terms of comprehensions. There is a great deal of confusion about list operators (is the result open, closed, when do overlapping constraints apply?). This confusion is eliminated by writing the equivalent as list of multiple comprehensions. Our intention is to remove these operators and have comprehensions (and later queries) be the only way in which lists can be concatenated or multiplied. Of course, cue fmt will help in the transition there as usual.

Embedded scalars

Just as it is possible to write

[ #a, #b ]

#a: 1
#b: 2

at the top of a CUE file, it is now also possible to have an embedded scalar along with definitions in any struct:

x: {
    [ #a, #b ]

    #a: 1
    #b: 2
}

Originally, the top-level construct was allowed to make CUE strictly compatible with JSON. Generalizing this to any struct makes the use of imported file objects consistent with normal structs. Moreover, it turns out that this construct can be quite useful overall.

Old CUE code is not affected by this change.

Semantics of [pattern]: value fields (no change after all)

The spec recently included a change that proposed to use the JSON schema semantics for pattern constraints ([k]: v), rather than the old CUE semantics. To aid in the transition, cue fmt rewrote these fields to be enclosed in curly braces, which mimics the old semantics under these planned rules.

It has now been decided to keep the old semantics and stop rewriting old uses.

As it turns out, the old semantics was used quite a bit and was actually the majority use case. The argument previously was that the JSON schema semantics was not easily expressible in terms of the old semantics, but that the old semantics is easy to simulate with the JSON schema semantics.

Instead we now intend to allow the JSON schema semantics additionally by writing [...k]: v. The nice thing here is that any pattern which is to be matched additionally, is now always preceded by a .... It also is a nicer generalization of the syntax.

But above all, it removes the need for a potentially painful transition.

Removed integer division operators

CUE allowed integer division and remainder per the div, mod, quo, and rem operators. These were relatively rarely used and complicated the CUE syntax, formatting and other parts of various subsystems.

Moreover, they made it lexically impossible to allow indexing of the form a.3 (instead of a[3]) down the road. We are not certain if we will allow this, but allowing it makes the syntax considerably more regular and consistent. With this change, we have the option to do so in a backwards compatible way.

These operators have now been replaced with builtins. As usual cue fmt will rewrite old uses to use the builtin for some time to come. It will use the __foo variant to avoid shadowing issues. These can be manually rewritten to remove the leading underscores if necessary.

Variable argument builtins

The model for builtins, which are still underspecified in the spec, is narrowed down more. It is now possible to define variable argument builtins.

The main reasons:

  1. Variable argument builtins are quite useful in being able to extend APIs in a backwards compatible way,
  2. It allows introducing some environment dependence in a clean, intuitive and nonetheless hermetic way.

The first target was pkg/path which now supports multiple OSes (see below).

Bug fixes

Errors in nested comprehension clauses were sometimes dropped, causing a failure in a nested if clause to be silently ignored. This has now been fixed.

Other fixes includes crashes related to let clauses and bytes interpolation.

Error messages

There are various areas where error messages have been improved.

cue fmt

There are various printing bugs that were fixed. Overall the stability of fmt should have been improved notably.

API

The API got a few fixes related to backwards compatibility and should now be more like v0.2.0 where this makes sense.

pkg/path

This package has now been extended to support multiple operating systems simultaneously. Most builtins have been extended to allow an additional optional argument to indicate the OS for which the builtin is to be applied. For instance:

path.Split("foo\bar", os)

will now return a different value depending on whether os is path.Windows or path.Unix. The os argument can also be omitted, in which case it defaults the existing behavior, which is identical to path.Unix.

Note that os will not default to the currently running OS. This would make the behavior non-hermetic. The idea is that the existing injection mechanism can be used (or a small extension of it to make this specific case easier) to inject an environment-specific value.

Backwards compatibility

The cue tool is now a bit more aggressive to panic if it encounters an internal inconsistency error. In return, it now allows the user to define a CUE_DEBUG=0 environment variable to disable this. In many cases the crashes can be ignored while still getting useful results.

Not using this option and filing bugs can help improve the stability of CUE though.

Main backwards incompatibility to note:

  • removed integer division operators (now builtins). This can be fixed by running cue fmt.
  • some of the bug fixes may cause CUE files that previously succeeded erroneously to fail now.

Changelog

e77ccb1 cmd/cue/cmd: don't bail for non-matching files
fecdd83 cmd/cue/cmd: ignore possible top-level tasks
bbe493a cmd/cue/cmd: more position information for field errors
fe73e8f cue/format: format let clause
110d0bf cue/format: pad floats with leading or trailing periods
fac5d2e cue/load: allow ./... to load all packages
555fb73 cue/parser: fix comment placement
1c904cc cue: allow multiple comprehensions in lists
8872b98 cue: fix PathCorrection tests
6c49cf0 cue: implement embedded scalars
d5177fd cue: map API values to more native ADT
b687b7f cue: remove aliases used for refactoring
19f6f3f doc/ref/spec.md: perpare for allowing a.4.f
f52a0ed doc/ref/spec.md: revert to the old semantics of [k]: v
76ea22c internal/core/adt: add Assert for debugging
37293f9 internal/core/adt: add methods for concreteness
41afc87 internal/core/adt: automatic renaming
2d568d3 internal/core/adt: better message for permanently non-concrete values
f62bfed internal/core/adt: clean up Builtin and Validator semantics
4db8ffb internal/core/adt: do list artihmetic with comprehensions
cbcb701 internal/core/adt: error message improvements
d0dd888 internal/core/adt: improve error message for optional fields
08a1652 internal/core/adt: introduce base value type
6de877a internal/core/adt: make progression check an assert
8b13752 internal/core/adt: move decimal logic into one place
b8c852d internal/core/adt: pass errors up in nested comprehensions
36f2051 internal/core/compile: add integer division builtins
9c6ded8 internal/core/compile: don't panic for incorrect labels
13e4af1 internal/core/compile: fix issue in and builtin
ff3e32f internal/core/convert: allow results of math to be integer
31896af internal/core/eval: add more tests for embedding
d49777f internal/core/eval: better incomplete error messages
55602d9 internal/core/eval: fix closedness issue for lists
9f1fec6 internal/core/eval: hoist vertex processing logic
cefe38b internal/core/eval: keep semantics for [K]: T
acc35f1 internal/core/eval: track fieldSets for data values
e7e27e0 internal/core/export: allow showing recursive errors.
039fb59 internal/core/export: fix bytes printing bug
2ea30a2 internal/core/export: fix crash with let comprehension export
16809ea internal/core/export: support byte interpolations
63594ef internal/core/runtime: assign first label to _.
0811662 internal/core/subsume: tests embeded scalars
a4d0928 internal/core/subsume: various bug fixes
136e51b internal/core: allow variable arg functions
e841714 internal/core: move equality checks from eval to adt
d174bc0 internal/cue/adt: remove use of Default in node
fa6308f internal/encoding: don't clobber files on crash
ec6f95d internal/filetpes: improve error messages
fb3b02e pkg/internal/builtin: use normal formatting for builtin test results
a50960c pkg/path/testdata: adapt Go implementation
a2692ef pkg/path/testdata: move files to minimize diffs
c77b9b0 pkg/path/testdata: stage filepath files from Go
85f8b46 pkg/path: activate OS-dependent version
64434c4 tools/fix: rewrite integer division
d5ced74 tools/flow: ignore faulty tasks outside the root

Assets 7
Loading