Skip to content

Commit

Permalink
Merge pull request #29 from NatLabs/one-shot
Browse files Browse the repository at this point in the history 
One shot Encoding and Decoding
  • Loading branch information
@tomijaga
tomijaga authored Jun 27, 2024
2 parents 964f2e8 + 74d8682 commit e94221d
Show file tree
Hide file tree
Showing 30 changed files with 10,152 additions and 2,212 deletions.
1,339 changes: 1,339 additions & 0 deletions Candid.md
View file

Large diffs are not rendered by default.

88 changes: 80 additions & 8 deletions bench.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,87 @@
#### mops version `2.2.0`
Benchmarking the performance with 10k calls

**Instructions**
Instructions

| | to_candid() | decode() | encode() |
| :---- | ----------: | ------------: | ------------: |
| Serde | 6_613_393 | 1_053_697_986 | 2_923_910_215 |
| | to_candid() | from_candid() | decode() | encode() |
| :---- | ----------: | ------------: | ------------: | ------------: |
| Serde | 6_847_934 | 20_275_957 | 1_053_547_402 | 2_000_462_160 |


**Heap**
Heap

| | to_candid() | decode() | encode() |
| :---- | ----------: | ---------: | --------: |
| Serde | 373_992 | 10_257_380 | 7_543_216 |
| | to_candid() | from_candid() | decode() | encode() |
| :---- | ----------: | ------------: | ---------: | ---------: |
| Serde | 373_360 | 397_012 | 10_425_876 | -1_620_764 |


#### one_shot: backward referencing

Instructions

| | decode() / too_candid() | encode() / from_candid() |
| :---------------------------------- | ----------------------: | -----------------------: |
| Serde 'mo:motoko_candid' lib | 1_053_963_520 | 2_000_335_920 |
| Motoko (to_candid(), from_candid()) | 6_851_133 | 20_284_651 |
| Serde: One Shot | 2_984 | 978_890_225 |
| Serde: One Shot sans type inference | 4_556 | 749_880_806 |


Heap

| | decode() / too_candid() | encode() / from_candid() |
| :---------------------------------- | ----------------------: | -----------------------: |
| Serde 'mo:motoko_candid' lib | 10_271_552 | -1_821_176 |
| Motoko (to_candid(), from_candid()) | 374_264 | 397_532 |
| Serde: One Shot | 8_904 | -10_726_840 |
| Serde: One Shot sans type inference | 8_904 | 28_404_880 |


#### one_show: optimized back reference and implemented forwad reference
Instructions

| | decode() / too_candid() | encode() / from_candid() |
| :------------------------------------- | ----------------------: | -----------------------: |
| Serde 'mo:motoko_candid' lib | 1_053_794_644 | 2_000_106_476 |
| Motoko (to_candid(), from_candid()) | 6_849_520 | 20_277_779 |
| Serde: One Shot Back Reference (BR) | 3_444 | 814_495_115 |
| Serde: One Shot BR sans type inference | 4_719 | 570_941_476 |
| Serde: One Shot Forward Reference (FR) | 5_861 | 1_002_895_852 |
| Serde: One Shot FR sans type inference | 7_067 | 759_206_807 |


Heap

| | decode() / too_candid() | encode() / from_candid() |
| :------------------------------------- | ----------------------: | -----------------------: |
| Serde 'mo:motoko_candid' lib | 10_271_024 | -1_831_772 |
| Motoko (to_candid(), from_candid()) | 373_872 | 397_408 |
| Serde: One Shot Back Reference (BR) | 8_904 | 14_959_180 |
| Serde: One Shot BR sans type inference | 8_904 | -7_202_408 |
| Serde: One Shot Forward Reference (FR) | 8_904 | -11_924_836 |
| Serde: One Shot FR sans type inference | 8_904 | -4_800_284 |

#### one_shot: br decoding

Instructions

| | decode() / too_candid() | encode() / from_candid() |
| :------------------------------------- | ----------------------: | -----------------------: |
| Motoko (to_candid(), from_candid()) | 6_852_599 | 20_277_307 |
| Serde 'mo:motoko_candid' lib | 1_054_078_235 | 2_000_977_097 |
| Serde: One Shot Back Reference (BR) | 283_341_316 | 814_744_565 |
| Serde: One Shot BR sans type inference | 259_040_591 | 576_643_675 |
| Serde: One Shot Forward Reference (FR) | 5_861 | 1_003_145_085 |
| Serde: One Shot FR sans type inference | 7_067 | 765_997_023 |


Heap

| | decode() / too_candid() | encode() / from_candid() |
| :------------------------------------- | ----------------------: | -----------------------: |
| Motoko (to_candid(), from_candid()) | 374_696 | 397_764 |
| Serde 'mo:motoko_candid' lib | 10_441_244 | -1_598_868 |
| Serde: One Shot Back Reference (BR) | 17_063_820 | -16_414_880 |
| Serde: One Shot BR sans type inference | 15_095_820 | -9_291_632 |
| Serde: One Shot Forward Reference (FR) | 8_904 | -11_919_024 |
| Serde: One Shot FR sans type inference | 8_904 | -4_795_636 |
161 changes: 143 additions & 18 deletions bench/serde.bench.mo
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ import Fuzz "mo:fuzz";
import Itertools "mo:itertools/Iter";

import Serde "../src";
import CandidEncoder "../src/Candid/Blob/Encoder";
import LegacyCandidEncoder "../src/Candid/Blob/Encoder.Legacy";
import CandidEncoderFR "../src/Candid/Blob/Encoder.ForwardReference";
import CandidDecoder "../src/Candid/Blob/Decoder";
import LegacyCandidDecoder "../src/Candid/Blob/Decoder.Legacy";

module {
public func init() : Bench.Bench {
Expand All @@ -18,9 +23,16 @@ module {
bench.name("Benchmarking Serde");
bench.description("Benchmarking the performance with 10k calls");

bench.rows(["Serde"]);
bench.rows([
"Serde: One Shot",
"Serde: One Shot sans type inference",
// "Serde 'mo:motoko_candid' lib",
"Motoko (to_candid(), from_candid())",

// "#Nat"
]);

bench.cols([
"to_candid()",
"decode()",
"encode()",
]);
Expand All @@ -44,21 +56,51 @@ module {
hex : Text;
};

// partial types for StoreItem
// as mo:motoko_candid is limited and throws errors on some complex types
type StoreItem = {
name : Text;
store : Text;
customer_reviews : [CustomerReview];
// available_sizes : AvailableSizes;
// color_options : [ColorOption];
available_sizes : AvailableSizes;
color_options : [ColorOption];
price : Float;
in_stock : Bool;
address : (Text, Text, Text, Text);
// contact : {
// email : Text;
// phone : ?Text;
// };
contact : {
email : Text;
phone : ?Text;
};
};

let CustomerReview = #Record([
("username", #Text),
("comment", #Text),
("rating", #Nat),
]);

let AvailableSizes = #Variant([("xs", #Null), ("s", #Null), ("m", #Null), ("l", #Null), ("xl", #Null)]);

let ColorOption = #Record([
("name", #Text),
("hex", #Text),
]);

let StoreItem : Serde.Candid.CandidType = #Record([
("name", #Text),
("store", #Text),
("customer_reviews", #Array(CustomerReview)),
("available_sizes", AvailableSizes),
("color_options", #Array(ColorOption)),
("price", #Float),
("in_stock", #Bool),
("address", #Tuple([#Text, #Text, #Text, #Text])),
("contact", #Record([
("email", #Text),
("phone", #Option(#Text)),
])),
]);

let candify_store_item = {
from_blob = func(blob : Blob) : StoreItem {
let ?c : ?StoreItem = from_candid (blob);
Expand Down Expand Up @@ -120,43 +162,126 @@ module {
};

let buffer = Buffer.Buffer<StoreItem>(limit);
let candid_blobs = Buffer.Buffer<Blob>(limit);
let candid_buffer = Buffer.Buffer<[Serde.Candid]>(limit);

for (i in Itertools.range(0, limit)) {
let item = new_item();
buffer.add(item);
};

let store_item_keys = ["name", "store", "customer_reviews", "username", "rating", "comment", "available_sizes", "xs", "s", "m", "l", "xl", "color_options", "name", "hex", "price", "in_stock", "address", "contact", "email", "phone"];
let StoreItemKeys = ["name", "store", "customer_reviews", "username", "rating", "comment", "available_sizes", "xs", "s", "m", "l", "xl", "color_options", "name", "hex", "price", "in_stock", "address", "contact", "email", "phone"];

bench.runner(
func(row, col) = switch (row, col) {
case ("Serde", "to_candid()") {
case ("Motoko (to_candid(), from_candid())", "encode()") {
for (i in Itertools.range(0, limit)) {
let item = buffer.get(i);
let candid = to_candid(item);
let candid = to_candid (item);
// candid_blobs.add(candid);
};
};
case ("Motoko (to_candid(), from_candid())", "decode()") {
for (i in Itertools.range(0, limit)) {
let blob = candid_blobs.get(i);
let ?store_item : ?StoreItem = from_candid (blob);
};
};
case ("Serde", "decode()") {

// case ("Serde 'mo:motoko_candid' lib", "decode()") {
// for (i in Itertools.range(0, limit)) {
// let item : StoreItem = buffer.get(i);
// let candid_blob = candify_store_item.to_blob(item);
// candid_blobs.add(candid_blob);
// let #ok(candid) = LegacyCandidDecoder.decode(candid_blob, StoreItemKeys, null);
// candid_buffer.add(candid);
// };
// };
// case ("Serde 'mo:motoko_candid' lib", "encode()") {
// for (i in Itertools.range(0, limit)) {
// let candid = candid_buffer.get(i);
// let res = LegacyCandidEncoder.encode(candid, null);
// let #ok(blob) = res;
// };
// };

case ("Serde: One Shot", "decode()") {
for (i in Itertools.range(0, limit)) {
let item = buffer.get(i);
let candid_blob = candify_store_item.to_blob(item);
let #ok(candid) = Serde.Candid.decode(candid_blob, store_item_keys, null);
candid_blobs.add(candid_blob);
let #ok(candid) = CandidDecoder.one_shot(candid_blob, StoreItemKeys, null);
candid_buffer.add(candid);
};
};
case ("Serde", "encode()") {
case ("Serde: One Shot", "encode()") {
for (i in Itertools.range(0, limit)) {
Debug.print("i = " # debug_show i);
let candid = candid_buffer.get(i);
let res = Serde.Candid.encode(candid, null);
Debug.print("res = " # debug_show res );
let res = CandidEncoder.one_shot(candid, null);
let #ok(blob) = res;
};
};

case ("Serde: One Shot sans type inference", "decode()") {
for (i in Itertools.range(0, limit)) {
let item = buffer.get(i);
let candid_blob = candify_store_item.to_blob(item);

let options = {
Serde.Candid.defaultOptions with types = ?[StoreItem]
};

let #ok(candid) = CandidDecoder.one_shot(candid_blob, StoreItemKeys, ?options);
// candid_buffer.add(candid);
};
};

case ("Serde: One Shot sans type inference", "encode()") {
for (i in Itertools.range(0, limit)) {
let candid = candid_buffer.get(i);

let options = {
Serde.Candid.defaultOptions with types = ?[StoreItem]
};
let res = CandidEncoder.one_shot(candid, ?options);
let #ok(blob) = res;
};
};

case ("Serde: One Shot Forward Reference (FR)", "decode()") {};
case ("Serde: One Shot Forward Reference (FR)", "encode()") {
for (i in Itertools.range(0, limit)) {
let candid = candid_buffer.get(i);
let res = CandidEncoderFR.one_shot(candid, null);
let #ok(blob) = res;
};
};

case ("Serde: One Shot FR sans type inference", "decode()") {};

case ("Serde: One Shot FR sans type inference", "encode()") {
for (i in Itertools.range(0, limit)) {
let candid = candid_buffer.get(i);

let options = {
Serde.Candid.defaultOptions with types = ?[StoreItem]
};
let res = CandidEncoderFR.one_shot(candid, ?options);
let #ok(blob) = res;
};
};

// case ("#Nat", "decode()"){
// for (i in Itertools.range(0, limit)) {
// let item = buffer.get(i);
// let candid_blob = candify_store_item.to_blob(item);
// let #ok(candid) = CandidDecoder.decode(candid_blob, StoreItemKeys, null);
// // candid_buffer.add(candid);
// };
// };
case (_, _) {
Debug.trap("Should be unreachable:\n row = \"" # debug_show row # "\" and col = \"" # debug_show col # "\"");
}
};
}
);

Expand Down
4 changes: 2 additions & 2 deletions mops.toml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "serde"
version = "2.3.1"
version = "3.0.0"
description = "A serialisation and deserialisation library for Motoko."
repository = "https://github.com/NatLabs/serde"
keywords = [ "json", "candid", "cbor", "urlencoded", "serialization" ]
Expand All @@ -11,7 +11,7 @@ base = "0.11.1"
itertools = "0.2.1"
candid = "1.0.2"
xtended-numbers = "0.2.1"
"json-float" = "https://github.com/NatLabs/json.mo#float@f3c8e7d418a7a8f2d6c0d7e2d276a0a82c2046ff"
json-float = "https://github.com/NatLabs/json.mo#float@f3c8e7d418a7a8f2d6c0d7e2d276a0a82c2046ff"
parser-combinators = "https://github.com/aviate-labs/parser-combinators.mo#v0.1.2@6a331bf78e9dcd7623977f06c8e561fd1a8c0103"
cbor = "0.1.3"
map = "9.0.1"
Expand Down
45 changes: 0 additions & 45 deletions package-set.dhall
View file

This file was deleted.

1 change: 1 addition & 0 deletions readme.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ Checkout the [usage guide](https://github.com/NatLabs/serde/blob/main/usage.md)
- Lack of specific syntax for conversion between `Blob`, `Principal`, and bounded `Nat`/`Int` types.
- Cannot deserialize Tuples as they are not candid types. They are just shorthands for records with unnamed fields. See https://forum.dfinity.org/t/candid-and-tuples/17800/7
- Floats are only recognised if they have a decimal point, e.g., `1.0` is a Float, but `1` is an `Int` / `Nat`.
- Only supports candid data types (i.e primitive and constructed types). Service and function reference types are not supported.

## Running Tests

Expand Down
Loading

0 comments on commit e94221d

Please sign in to comment.