From d327618388818ee6979c7e5cdc3fbda8de636c43 Mon Sep 17 00:00:00 2001 From: Andreas Rossberg Date: Wed, 3 Nov 2021 09:45:01 +0100 Subject: [PATCH 1/6] Add support for bound function arguments --- spec/Candid.md | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/spec/Candid.md b/spec/Candid.md index 576ac1c6..87d56e14 100644 --- a/spec/Candid.md +++ b/spec/Candid.md @@ -464,7 +464,10 @@ A *function reference* is described by its function type. For example, they allo ::= func | ... ``` -##### Example +A function reference may be represented by a simple closure, encapsulating a prefix of arguments that have previously been bound. These hidden arguments do not appear in the function type, but will be forwarded implicitly when the function itself is invoked. + + +##### Examples ``` type engine = service { @@ -472,6 +475,15 @@ type engine = service { } ``` +``` +type wallet = service { + topup : (amount : nat) -> (); + forward : (call : () -> ()) -> (); +} +``` +In the latter example, the `call` parameter is assumed to be a closure encapsulating a call to another service (including bound arguments) that the wallet executes on its own caller's behalf by invoking the function. + + #### Principal References A *principal reference* points to an identity, such as a canister or a user. Through this, we can authenticate or authorize other services or users. Because the type constructor takes no arguments, it is classified as a _primitive_ type. @@ -571,9 +583,9 @@ The types of these values are assumed to be known from context, so the syntax do ::= = ::= - | service (canister URI) - | func . (canister URI and message name) - | principal (principal URI) + | principal (principal URI) + | service (canister URI) + | func . ( ,* )? (canister URI, message name, and possibly bound arguments) ::= ( ,* ) @@ -1064,7 +1076,7 @@ Most Candid values are self-explanatory, except for references. There are two fo Likewise, there are two forms of Candid values for function references: * `ref(r)` indicates an opaque reference, understood only by the underlying system. -* `pub(s,n)`, indicates the public method name `n` of the service referenced by `s`. +* `pub(s,n,v*:t*)`, indicates the public method name `n` of the service referenced by `s`, possibly followed by a list of type-annotated bound argument values. #### Notation @@ -1188,11 +1200,15 @@ M : -> -> i8* M(ref(r) : service ) = i8(0) M(id(v*) : service ) = i8(1) M(v* : vec nat8) -M(ref(r) : func ) = i8(0) -M(pub(s,n) : func ) = i8(1) M(s : service {}) M(n : text) +M(ref(r) : func ) = i8(0) +M(pub(s,n) : func ) = i8(1) M(s : service {}) M(n : text) +M(pub(s,n,v*:t*) : func ) = i8(2) M(s : service {}) M(n : text) M*(v* : t*) M(ref(r) : principal) = i8(0) M(id(v*) : principal) = i8(1) M(v* : vec nat8) + +M* : * -> * -> i8* +M*(v^N : ^N) = leb128(N) M(v : )^N ``` @@ -1218,10 +1234,13 @@ R((k,v) : k:) = R(v : ) R : -> -> * R(ref(r) : service ) = r R(id(b*) : service ) = . -R(ref(r) : func ) = r -R(pub(s,n) : func ) = . +R(ref(r) : func ) = r +R(pub(s,n,v*:t*) : func ) = R*(v* : t*) R(ref(r) : principal) = r R(id(b*) : principal) = . + +R* : * -> * -> * +R*(v^N : ^N) = R(v : )^N ``` Note: From 3aa81a109f3e96baf1558d21074a9028a38d9ab2 Mon Sep 17 00:00:00 2001 From: Andreas Rossberg Date: Wed, 3 Nov 2021 18:08:08 +0100 Subject: [PATCH 2/6] Comments --- spec/Candid.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/Candid.md b/spec/Candid.md index 87d56e14..60e2518c 100644 --- a/spec/Candid.md +++ b/spec/Candid.md @@ -478,7 +478,7 @@ type engine = service { ``` type wallet = service { topup : (amount : nat) -> (); - forward : (call : () -> ()) -> (); + forward : (call : func () -> ()) -> (); } ``` In the latter example, the `call` parameter is assumed to be a closure encapsulating a call to another service (including bound arguments) that the wallet executes on its own caller's behalf by invoking the function. @@ -573,6 +573,7 @@ The types of these values are assumed to be known from context, so the syntax do | | true | false | null + | principal (principal URI) ::= | opt @@ -583,7 +584,6 @@ The types of these values are assumed to be known from context, so the syntax do ::= = ::= - | principal (principal URI) | service (canister URI) | func . ( ,* )? (canister URI, message name, and possibly bound arguments) From c5979f8b9121f14fe62252e2b27c53eb6f4e6d53 Mon Sep 17 00:00:00 2001 From: Andreas Rossberg Date: Fri, 5 Nov 2021 10:40:15 +0100 Subject: [PATCH 3/6] Comments --- spec/Candid.md | 98 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 69 insertions(+), 29 deletions(-) diff --git a/spec/Candid.md b/spec/Candid.md index 60e2518c..ec6baf0d 100644 --- a/spec/Candid.md +++ b/spec/Candid.md @@ -81,10 +81,10 @@ This is a summary of the grammar proposed: | | bool | text + | principal | null | reserved | empty - | principal ::= | nat | nat8 | nat16 | nat32 | nat64 @@ -99,6 +99,7 @@ This is a summary of the grammar proposed: ::= | func + | closure | service ::= | @@ -280,6 +281,15 @@ Text strings are represented by the type `text` and consist of a sequence of Uni ``` **Note:** The `text` type is distinguished from `vec nat8` (a UTF-8 string) or `vec nat32` (a sequence of code points) in order to allow bindings to map it to a suitable string type, and enable the binary format to select an efficient internal representation independently. + +#### Principal + +A *principal* points to an identity, such as a canister or a user. Through this, we can authenticate or authorize other services or users. Because the type constructor takes no arguments, it is classified as a _primitive_ type. + +``` + ::= ... | principal | ... +``` + #### Null The type `null` has exactly one value (the *null* value) and therefore carries no information. It can e.g. be used as a placeholder for optional fields that ought to be added to a record in future upgrades, or for *variant cases* that do not need any value, see below. @@ -464,10 +474,7 @@ A *function reference* is described by its function type. For example, they allo ::= func | ... ``` -A function reference may be represented by a simple closure, encapsulating a prefix of arguments that have previously been bound. These hidden arguments do not appear in the function type, but will be forwarded implicitly when the function itself is invoked. - - -##### Examples +##### Example ``` type engine = service { @@ -475,22 +482,28 @@ type engine = service { } ``` + +#### Closure References + +A *closure reference* is also described by its function type. Like function references, they allow passing callbacks to other functions, but they may additionally encapsulate a prefix of arguments that have previously been bound. These hidden arguments do not appear in the function type, but will be forwarded implicitly when the function closure itself is invoked. + ``` -type wallet = service { - topup : (amount : nat) -> (); - forward : (call : func () -> ()) -> (); -} + ::= ... | closure | ... ``` -In the latter example, the `call` parameter is assumed to be a closure encapsulating a call to another service (including bound arguments) that the wallet executes on its own caller's behalf by invoking the function. +Note: Closures are more general than functions, so in most cases, a service should allow closures as arguments instead of plain functions. -#### Principal References -A *principal reference* points to an identity, such as a canister or a user. Through this, we can authenticate or authorize other services or users. Because the type constructor takes no arguments, it is classified as a _primitive_ type. +##### Example ``` - ::= ... | principal | ... +type wallet = service { + topup : (amount : nat) -> (); + forward : (call : closure () -> ()) -> (); +} ``` +In the latter example, the `call` parameter is assumed to be a closure encapsulating a call to another service (including bound arguments) that the wallet executes on its own caller's behalf by invoking the function. + ### Type Definitions @@ -584,8 +597,9 @@ The types of these values are assumed to be known from context, so the syntax do ::= = ::= - | service (canister URI) - | func . ( ,* )? (canister URI, message name, and possibly bound arguments) + | service (canister URI) + | func . (canister URI, message name) + | closure . ( ,* ) (canister URI, message name, bound arguments) ::= ( ,* ) @@ -865,18 +879,21 @@ variant { : ; ;* } <: variant { : ; *Note:* By virtue of the rules around `opt` above, it is possible to evolve and extend variant types that also occur in outbound position (i.e., are used both as function results and function parameters) by *adding* tags to variants, provided the variant itself is optional (e.g. `opt variant { 0 : nat; 1 : bool } <: opt variant { 1 : bool }`). Any party not aware of the extension will treat the new case as `null`. -#### Functions +#### Functions and Closures For a specialised function, any parameter type can be generalised and any result type specialised. Moreover, arguments can be dropped while results can be added. That is, the rules mirror those of tuple-like records, i.e., they are ordered and can only be extended at the end. + +Closures have the same subtyping rules as functions. In addition, any function can be treated as a trivial closure. ``` +kind1 = kind2 \/ kind1 = func record { (N1' : );* } <: record { (N1 : );* } record { (N2 : );* } <: record { N2' : );* } ------------------------------------------------------------------------------------------------------------------- -func ( ,* ) -> ( ,* ) * <: func ( ,* ) -> ( ,* ) * +kind1 ( ,* ) -> ( ,* ) * <: kind2 ( ,* ) -> ( ,* ) * ``` where `NI*` is the `` sequence `1`..`|*|`, respectively. -Viewed as sets, the annotations on the functions must be equal. +Viewed as sets, the annotations on the function type must be equal. #### Services @@ -987,6 +1004,12 @@ C[service <: service ](service ) = service C[principal <: principal](principal ) = principal ``` +However, functions can be converted into closures with an empty list of bound arguments: +``` +C[func <: closure ](func .) = clos(func ., .) +``` + + #### Tuple types Whole argument and result sequences are coerced with the same rules as tuple-like records. In particular, extra arguments are ignored, and optional parameters read as as `null` if the argument is missing or fails to coerce: @@ -1070,13 +1093,18 @@ Serialisation is defined by three functions `T`, `M`, and `R` given below. Most Candid values are self-explanatory, except for references. There are two forms of Candid values for service references and principal references: -* `ref(r)` indicates an opaque reference, understood only by the underlying system. +* `ref(r)`, indicates an opaque reference, understood only by the underlying system. * `id(b)`, indicates a transparent reference to a service addressed by the blob `b`. Likewise, there are two forms of Candid values for function references: -* `ref(r)` indicates an opaque reference, understood only by the underlying system. -* `pub(s,n,v*:t*)`, indicates the public method name `n` of the service referenced by `s`, possibly followed by a list of type-annotated bound argument values. +* `ref(r)`, indicates an opaque reference, understood only by the underlying system. +* `pub(s,n)`, indicates the public method name `n` of the service referenced by `s`. + +Finally, a closure pairs a function reference value `f` with a list of bound argument values: + +* `clos(f,v*:t*)`, where `f` is one of the above, and binds the argument values `v*`, annotated with their respective type. + #### Notation @@ -1131,7 +1159,10 @@ T : -> i8* T(func (*) -> (*) *) = sleb128(-22) T*(*) T*(*) T*(*) // 0x6a T(service {*}) = - sleb128(-23) T*(*) // 0x69 + sleb128(-23) T*(*) // 0x69 +T(closure (*) -> (*) *) = + sleb128(-26) leb128(|T*(*) T*(*) T*(*)|) // 0x66 + T*(*) T*(*) T*(*) T : -> i8* T(:) = leb128(|utf8()|) i8*(utf8()) I() @@ -1200,15 +1231,21 @@ M : -> -> i8* M(ref(r) : service ) = i8(0) M(id(v*) : service ) = i8(1) M(v* : vec nat8) -M(ref(r) : func ) = i8(0) -M(pub(s,n) : func ) = i8(1) M(s : service {}) M(n : text) -M(pub(s,n,v*:t*) : func ) = i8(2) M(s : service {}) M(n : text) M*(v* : t*) +M(ref(r) : func ) = i8(0) +M(pub(s,n) : func ) = i8(1) M(s : service {}) M(n : text) +M(clos(f,v*:t*) : closure ) = + leb128(|i8(2) M(f : func ) TM*(v* : t*)|) + leb128(|R(f : func ) R*(v* : t*)|) + i8(2) M(f : func ) TM*(v* : t*) M(ref(r) : principal) = i8(0) M(id(v*) : principal) = i8(1) M(v* : vec nat8) -M* : * -> * -> i8* -M*(v^N : ^N) = leb128(N) M(v : )^N +TM : -> -> i8* +TM(v : ) = T() M(v : ) + +TM* : * -> * -> i8* +TM*(v^N : ^N) = leb128(N) TM(v : )^N ``` @@ -1235,7 +1272,8 @@ R : -> -> * R(ref(r) : service ) = r R(id(b*) : service ) = . R(ref(r) : func ) = r -R(pub(s,n,v*:t*) : func ) = R*(v* : t*) +R(pub(s,n) : func ) = R*(v* : t*) +R(clos(f,v*:t*) : closure ) = R(f : func ) R*(v* : t*) R(ref(r) : principal) = r R(id(b*) : principal) = . @@ -1286,7 +1324,9 @@ Deserialisation uses the following mechanism for robustness towards future exten * A serialised type may be headed by an opcode other than the ones defined above (i.e., less than -24). Any such opcode is followed by an LEB128-encoded count, and then a number of bytes corresponding to this count. A type represented that way is called a *future type*. -* A value corresponding to a future type is called a *future value*. It is represented by two LEB128-encoded counts, *m* and *n*, followed by a *m* bytes in the memory representation M and accompanied by *n* corresponding references in R. +* A serialised type may be headed by an other than -1 to -24 . Any such opcode is followed by an LEB128-encoded count, and then a number of bytes corresponding to this count. A type represented that way is called a *future type*. + +Closure types are the only future type so far. These measures allow the serialisation format to be extended with new types in the future, as long as their representation and the representation of the corresponding values include a length prefix matching the above scheme, and thereby allowing an older deserialiser not understanding them to skip over them. The subtyping rules ensure that upgradability is maintained in this situation, i.e., an old deserialiser has no need to understand the encoded data. From bf7f1a5abe836e2a2c180b4914eb944a9f4e3e8d Mon Sep 17 00:00:00 2001 From: Andreas Rossberg Date: Mon, 8 Nov 2021 11:41:44 +0100 Subject: [PATCH 4/6] Comments from Joachim --- spec/Candid.md | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/spec/Candid.md b/spec/Candid.md index ec6baf0d..c92b8ca0 100644 --- a/spec/Candid.md +++ b/spec/Candid.md @@ -1006,7 +1006,7 @@ C[principal <: principal](principal ) = principal However, functions can be converted into closures with an empty list of bound arguments: ``` -C[func <: closure ](func .) = clos(func ., .) +C[func <: closure ](f) = clos(f, .) ``` @@ -1161,8 +1161,7 @@ T(func (*) -> (*) *) = T(service {*}) = sleb128(-23) T*(*) // 0x69 T(closure (*) -> (*) *) = - sleb128(-26) leb128(|T*(*) T*(*) T*(*)|) // 0x66 - T*(*) T*(*) T*(*) + FT(-26, T*(*) T*(*) T*(*)) // 0x66 T : -> i8* T(:) = leb128(|utf8()|) i8*(utf8()) I() @@ -1174,6 +1173,7 @@ T(oneway) = i8(2) T* : * -> i8* T*(^N) = leb128(N) T()^N ``` +The meta-function `FT` constructs a backwards-compatible encoding for [future types](#deserialisation-of-future-types). Every nested type is encoded as either a primitive type, via the negative op-code, or an index into a list of *type definitions*, via a positive number. This allows for recursive types and sharing of types occuring multiple times: @@ -1234,19 +1234,21 @@ M(id(v*) : service ) = i8(1) M(v* : vec nat8) M(ref(r) : func ) = i8(0) M(pub(s,n) : func ) = i8(1) M(s : service {}) M(n : text) M(clos(f,v*:t*) : closure ) = - leb128(|i8(2) M(f : func ) TM*(v* : t*)|) - leb128(|R(f : func ) R*(v* : t*)|) - i8(2) M(f : func ) TM*(v* : t*) + FM( + i8(2) M(f : func ) TM*(v* : t*), + R(f : func ) R*(v* : t*) + ) M(ref(r) : principal) = i8(0) M(id(v*) : principal) = i8(1) M(v* : vec nat8) TM : -> -> i8* -TM(v : ) = T() M(v : ) +TM(v : ) = I() M(v : ) TM* : * -> * -> i8* TM*(v^N : ^N) = leb128(N) TM(v : )^N ``` +The meta-function `FM` constructs a backwards-compatible encoding for values of for [future types](#deserialisation-of-future-types). #### References @@ -1322,9 +1324,17 @@ Deserialisation at an expected type sequence `(,*)` proceeds by Deserialisation uses the following mechanism for robustness towards future extensions: -* A serialised type may be headed by an opcode other than the ones defined above (i.e., less than -24). Any such opcode is followed by an LEB128-encoded count, and then a number of bytes corresponding to this count. A type represented that way is called a *future type*. - * A serialised type may be headed by an other than -1 to -24 . Any such opcode is followed by an LEB128-encoded count, and then a number of bytes corresponding to this count. A type represented that way is called a *future type*. + ``` + FT : i32 -> i8* -> i8* + FT(n, b*) = sleb128(n) leb128(|b*|) b* + ``` + +* A value corresponding to a future type is called a *future value*. It is represented by two LEB128-encoded counts, *m* and *n*, followed by a *m* bytes in the memory representation M and accompanied by *n* corresponding references in R. + ``` + FM : i8* -> ref* -> i8* + FM(b*, r*) = leb128(|b*|) leb128(r*) b* + ``` Closure types are the only future type so far. From 97ea0e771183dca07b888969d5ea15e70b277ed2 Mon Sep 17 00:00:00 2001 From: Andreas Rossberg Date: Tue, 9 Nov 2021 18:47:34 +0100 Subject: [PATCH 5/6] Update spec/Candid.md Co-authored-by: Joachim Breitner --- spec/Candid.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/Candid.md b/spec/Candid.md index c92b8ca0..04d6082e 100644 --- a/spec/Candid.md +++ b/spec/Candid.md @@ -1333,7 +1333,7 @@ Deserialisation uses the following mechanism for robustness towards future exten * A value corresponding to a future type is called a *future value*. It is represented by two LEB128-encoded counts, *m* and *n*, followed by a *m* bytes in the memory representation M and accompanied by *n* corresponding references in R. ``` FM : i8* -> ref* -> i8* - FM(b*, r*) = leb128(|b*|) leb128(r*) b* + FM(b*, r*) = leb128(|b*|) leb128(|r*|) b* ``` Closure types are the only future type so far. From ffef920eebd7d328c868c6bac76520a77c61656b Mon Sep 17 00:00:00 2001 From: Andreas Rossberg Date: Mon, 22 Nov 2021 09:51:25 +0100 Subject: [PATCH 6/6] Update spec/Candid.md Co-authored-by: Claudio Russo --- spec/Candid.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/Candid.md b/spec/Candid.md index 04d6082e..b41a6c29 100644 --- a/spec/Candid.md +++ b/spec/Candid.md @@ -1248,7 +1248,7 @@ TM(v : ) = I() M(v : ) TM* : * -> * -> i8* TM*(v^N : ^N) = leb128(N) TM(v : )^N ``` -The meta-function `FM` constructs a backwards-compatible encoding for values of for [future types](#deserialisation-of-future-types). +The meta-function `FM` constructs a backwards-compatible encoding for values of [future types](#deserialisation-of-future-types). #### References