chore: Upgrade RTS Dependencies #4677

luc-blaeser · 2024-08-29T09:04:46Z

Upgrading Motoko RTS dependencies:

LLVM 18.1.8
Rust 1.82.0

Special aspects:

Support new dylink.0 instead of dylink.
Support GOT.mem for RTS in the Motoko linker.
Adjusting the RTS code in line with the latest Rust features/rules.
No longer needing emscripten: Use custom Rust targets .
Get rid of emscripten by using custom Rust targets for generating PIC-relocatable Wasm.
Get rid of musl by using Rust builtins instead.
Float formatting changes due to removal of musl: The output format of NaN and infinity has changed. Moreover, one particular output format (hexadecimal float) is no longer supported. Updating the documentation and base library is needed.
Optimization: Use bulk-memory operations because memcpy of Rust is slower than musl.

Note:

If your nix-shell crashes with a segmentation fault with this PR, you need to upgrade nix. For this, you need to manually uninstall and re-install nix.

TODO:

Merge Base Library Adjustments for Latest Motoko RTS motoko-base#589: Then we can change the motoko-base dependency back to next-moc.

github-actions · 2024-08-29T11:17:46Z

Comparing from b7fa639 to 6572974:
In terms of gas, 3 tests regressed, 2 tests improved and the mean change is -0.0%.
In terms of size, 5 tests improved and the mean change is -1.6%.

rts/Makefile

* Patch Rust for shared Wasm libraries * Get rid of `musl` * Further simplify Wasm linking * Update benchmark results * No longer need `wasm-ld` patch * Update test result * Add distinction for Linux Ocaml build * Revert "No longer need `wasm-ld` patch" This reverts commit 3d0f34e. * Revert "Update test result" This reverts commit 8fc5c08. * Use patched wasm-ld during Rust build * Table offset must be at least 1 for Rust RTS * Adjust linker test result * Use bulk memory operations * Do not need `wasm-lld` patch with shared targets * Adjust expected test result * Adjust nix build script * Fix nix build script

luc-blaeser · 2024-10-09T14:12:23Z

Size compression is no longer needed for the specific Rust target backends.
Instead, we can again optimize for speed:

This is the latest perf diff on CI (locally repeated):

Comparing from b7fa63936cf708952deae0dd0b0429faa1baa119 to a10ab53a1ec3e5e48dbce36d6c7dd8547503c85c:
In terms of gas, 3 tests regressed, 2 tests improved and the mean change is -0.0%.
In terms of size, 5 tests improved and the mean change is -1.6%.

luc-blaeser · 2024-10-09T16:13:22Z

According to the GC benchmark, it seems that this upgrade/cleanup even improves performance:

Mode	Before	Now	Improvement
Classical (copying GC)	2.33e10	2.29e10	1.7%
Classical (incremental GC)	2.39e10	2.35e10	1.7%
EOP	2.54e0	2.50e10	1.6%

luc-blaeser · 2024-10-09T16:30:41Z

nix/sources.json

-        "branch": "next-moc",
+        "branch": "pull/589/head",
        "description": "The Motoko base library",
        "homepage": "",
        "owner": "dfinity",
        "repo": "motoko-base",
-        "rev": "21a80ab250501a5de3365114abb242559573fecc",
-        "sha256": "0zm60azafdfczdiqkqs1fwgxf3crb2bk0vs7y09rvnwkqgqkxxw8",
+        "rev": "1bece926bfc417db5661c66d8617f6b09ab06b95",
+        "sha256": "02i8s4ybjmbbf72jclpvqcsxh4dypgsxqsagiga64jqyprmqzvlr",
        "type": "tarball",
-        "url": "https://github.com/dfinity/motoko-base/archive/21a80ab250501a5de3365114abb242559573fecc.tar.gz",
+        "url": "https://github.com/dfinity/motoko-base/archive/1bece926bfc417db5661c66d8617f6b09ab06b95.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"


This is to be changed back to next-moc once dfinity/motoko-base#589 is merged.

Let's not forget this

ggreif · 2024-11-25T12:59:12Z

nix/default.nix

@@ -78,7 +78,7 @@ let
                };
              };

-              # No testing of atdgen, as it pulls in python stuff, tricky on musl
+              # No testing of atdgen, as it pulls in python stuff
              atdgen = super.ocamlPackages.atdgen.overrideAttrs { doCheck = false; };


this could be doCheck = true now?

or rather leave it uncustomized?

ggreif · 2024-11-25T13:00:17Z

nix/default.nix

        in rec {
          rustc-nightly = rust-channel.rust.override {
            targets = [
-               "wasm32-unknown-emscripten"


oh, why were these here?

ggreif · 2024-11-25T13:02:17Z

default.nix

-    # pulls in clang (wrapped) and clang-13 (unwrapped)
-    llvmPackages_13.clang
-    # pulls in wasm-ld
-    llvmPackages_13.lld


this package now merged into bintools? We do use wasm-ld-18 below...

rts/motoko-rts/Cargo.toml

ggreif · 2024-11-25T13:07:15Z

rts/motoko-rts/src/float.rs


-use motoko_rts_macros::{classical_persistence, enhanced_orthogonal_persistence};
+// The meaning of the `mode` parameter is documented in motoko-base, function Float.format()


ggreif · 2024-11-25T13:11:21Z

src/linking/linkModule.ml

+  List.filter (fun import -> not (is_got import.it.module_name)) imports
+


Suggested change

List.filter (fun import -> not (is_got import.it.module_name)) imports

List.filter (fun import -> not (is_got import.it.module_name)) imports

added newline removed

ggreif · 2024-11-25T13:13:45Z

src/wasm-exts/customModuleEncode.ml

@@ -915,7 +916,7 @@ let encode (em : extended_module) =
      patch s (p + 1) (lsb (n lsr 8));
      patch s (p + 2) (lsb (n lsr 16));
      patch s (p + 3) (lsb (n lsr 24))
-    let dw_patches = ref Fun.id
+    let dw_patches = ref (fun i -> i)


heh, why this?

test/run/float-fmt.mo

ggreif

LGTM! Heroic work on the dylink0 stuff. Being stuck on clang-13 gave me nightmares...
Nice to see the cycle improvements too. Getting rid of musl means one less wheel to maintain.

Good work! Added a few Qs...

crusso

Great work! Not sure I fully understand the intricacies so 🤞

crusso · 2024-11-25T15:53:40Z

nix/sources.json

-        "branch": "next-moc",
+        "branch": "pull/589/head",
        "description": "The Motoko base library",
        "homepage": "",
        "owner": "dfinity",
        "repo": "motoko-base",
-        "rev": "21a80ab250501a5de3365114abb242559573fecc",
-        "sha256": "0zm60azafdfczdiqkqs1fwgxf3crb2bk0vs7y09rvnwkqgqkxxw8",
+        "rev": "1bece926bfc417db5661c66d8617f6b09ab06b95",
+        "sha256": "02i8s4ybjmbbf72jclpvqcsxh4dypgsxqsagiga64jqyprmqzvlr",
        "type": "tarball",
-        "url": "https://github.com/dfinity/motoko-base/archive/21a80ab250501a5de3365114abb242559573fecc.tar.gz",
+        "url": "https://github.com/dfinity/motoko-base/archive/1bece926bfc417db5661c66d8617f6b09ab06b95.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"


Let's not forget this

crusso · 2024-11-25T16:01:33Z

rts/motoko-rts/src/allocator.rs

@@ -41,33 +40,3 @@ unsafe impl GlobalAlloc for EphemeralAllocator {

 #[global_allocator]
 static ALLOCATOR: EphemeralAllocator = EphemeralAllocator;
-


Was this stuff need for musl and that's why we can remove it?

crusso · 2024-11-26T11:24:51Z

src/linking/linkModule.ml

-  if imports = [] then
-    m
+  (* Computes the absolute address of the GOT.mem data pointer *)
+  let data_pointer exported_global_index = if uses_memory64 m then


Not that you should do this now, but I wonder if it would be possible to just bump the (mutable?) globals by the offset using an initialization function, rather than recomputing the absolute address at each read. Another advantage, apart from perf, would be that any debug info would remain valid (rather than disturbing the instruction offset in the code.

crusso · 2024-11-26T11:56:53Z

src/linking/linkModule.ml

+  For each `GOT.mem`, we replace the GOT accesses in the AST by code that computes 
+  the absolute address of the data. This is the library's memory base plus the 
+  relative data offset stored in the global as determined in the first step.  
+  The `GOT.mem` import is replaced by a dummy global that only serves to maintain 


I wonder if the dummy global could instead hold the absolute/resolved address, and we replace references to the import by references to the resolved global? But I don't fully understand this stuff..

luc-blaeser added 3 commits August 29, 2024 10:29

Upgrade RTS dependencies

4984cd5

Shrink RTS size

fdd2ff2

Link 64-bit RTS Rust code to one shared object file

9153fca

This was referenced Aug 29, 2024

EOP: Rust Upgrade for RTS #4648

Closed

Experiment: RTS Size Optimizations #4661

Closed

luc-blaeser self-assigned this Aug 29, 2024

luc-blaeser added the dependencies Pull requests that update a dependency file label Aug 29, 2024

luc-blaeser marked this pull request as ready for review August 29, 2024 11:57

luc-blaeser mentioned this pull request Aug 29, 2024

Base Library Adjustments for Latest Motoko RTS dfinity/motoko-base#589

Open

Update Motoko base dependency

a6d1305

luc-blaeser requested a review from ggreif August 29, 2024 12:58

luc-blaeser requested a review from crusso September 9, 2024 08:19

crusso reviewed Sep 10, 2024

View reviewed changes

rts/Makefile Outdated Show resolved Hide resolved

luc-blaeser and others added 4 commits October 8, 2024 14:44

Merge branch 'master' into luc/upgrade-rts-dependencies

04e04e6

Adjust Motoko base dependency

af74f86

Skip size optimization

a10ab53

luc-blaeser requested a review from crusso October 9, 2024 16:17

luc-blaeser commented Oct 9, 2024

View reviewed changes

luc-blaeser added 5 commits October 9, 2024 18:36

Cleanup config.toml

7aa01e7

Cleanup TODO

f8a970d

Adjust format

6572974

Format back to original

323d52e

Reformat

923086b

ggreif reviewed Nov 25, 2024

View reviewed changes

rts/motoko-rts/Cargo.toml Show resolved Hide resolved

ggreif reviewed Nov 25, 2024

View reviewed changes

test/run/float-fmt.mo Show resolved Hide resolved

ggreif approved these changes Nov 25, 2024

View reviewed changes

crusso approved these changes Nov 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Upgrade RTS Dependencies #4677

chore: Upgrade RTS Dependencies #4677

luc-blaeser commented Aug 29, 2024 •

edited

Loading

github-actions bot commented Aug 29, 2024 •

edited

Loading

luc-blaeser commented Oct 9, 2024

luc-blaeser commented Oct 9, 2024

luc-blaeser Oct 9, 2024

crusso Nov 25, 2024

ggreif Nov 25, 2024

ggreif Nov 25, 2024 •

edited

Loading

ggreif Nov 25, 2024

ggreif Nov 25, 2024

ggreif Nov 25, 2024

ggreif Nov 25, 2024 •

edited

Loading

ggreif Nov 25, 2024

ggreif left a comment

crusso left a comment •

edited

Loading

crusso Nov 25, 2024

crusso Nov 25, 2024

crusso Nov 26, 2024

crusso Nov 26, 2024


		use motoko_rts_macros::{classical_persistence, enhanced_orthogonal_persistence};
		// The meaning of the `mode` parameter is documented in motoko-base, function Float.format()

		List.filter (fun import -> not (is_got import.it.module_name)) imports

		@@ -41,33 +40,3 @@ unsafe impl GlobalAlloc for EphemeralAllocator {

		#[global_allocator]
		static ALLOCATOR: EphemeralAllocator = EphemeralAllocator;

chore: Upgrade RTS Dependencies #4677

Are you sure you want to change the base?

chore: Upgrade RTS Dependencies #4677

Conversation

luc-blaeser commented Aug 29, 2024 • edited Loading

github-actions bot commented Aug 29, 2024 • edited Loading

luc-blaeser commented Oct 9, 2024

luc-blaeser commented Oct 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggreif Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggreif Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggreif left a comment

Choose a reason for hiding this comment

crusso left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luc-blaeser commented Aug 29, 2024 •

edited

Loading

github-actions bot commented Aug 29, 2024 •

edited

Loading

ggreif Nov 25, 2024 •

edited

Loading

ggreif Nov 25, 2024 •

edited

Loading

crusso left a comment •

edited

Loading