Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow support for Wasm modules #402

Open
wants to merge 82 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
4af042c
rename 'AssertedModuleType' to 'RequestedModuleType'
bartlomieju Dec 29, 2023
394c674
more renames
bartlomieju Dec 29, 2023
3404e94
test that creating a custom module fails
bartlomieju Dec 29, 2023
b0197b7
Merge branch 'main' into request_module_type2
bartlomieju Dec 29, 2023
55eb3aa
custom module evaluation cb
bartlomieju Dec 29, 2023
f6106ed
wire up the callback
bartlomieju Dec 29, 2023
3e88246
something is working, but module type is broken
bartlomieju Dec 29, 2023
907f640
fmt
bartlomieju Dec 29, 2023
4abc9f3
text import working too
bartlomieju Dec 29, 2023
444d7b4
and url import work too
bartlomieju Dec 29, 2023
333a934
a bit of cleanup
bartlomieju Dec 29, 2023
bbc40db
Divy's suggestion
bartlomieju Jan 1, 2024
611401d
Add ModuleSourceBytes enum
bartlomieju Jan 3, 2024
b1dadd8
support top-level WASM imports
bartlomieju Jan 3, 2024
f4d9366
add dprint plugin
bartlomieju Jan 3, 2024
498d018
Merge branch 'main' into request_module_type2
bartlomieju Jan 9, 2024
cedff3f
Merge branch 'main' into request_module_type2
bartlomieju Jan 11, 2024
3d61bd9
this is how it should look like
bartlomieju Jan 11, 2024
21bd425
keep so I don't forget
bartlomieju Jan 11, 2024
b81c0fd
Merge branch 'main' into request_module_type2
bartlomieju Jan 11, 2024
216ec1f
Merge branch 'main' into request_module_type2
bartlomieju Jan 11, 2024
28a0d97
dedup
bartlomieju Jan 11, 2024
360271d
Merge branch 'main' into request_module_type2
bartlomieju Jan 12, 2024
902fae4
fix after merge
bartlomieju Jan 12, 2024
c28fd3c
remove url imports
bartlomieju Jan 12, 2024
8c98195
deny json imports if not json
bartlomieju Jan 12, 2024
9573e6f
wasm-module import to show PoC of exported members of WASM module
bartlomieju Jan 13, 2024
ba2b85f
lint
bartlomieju Jan 13, 2024
cf9a119
some cleanup, add Deno.core.isWasmModuleObject()
bartlomieju Jan 13, 2024
7e7535a
add CustomModuleEvaluationKind enum
bartlomieju Jan 13, 2024
7dbc4b1
no ModuleMap in CustomModuleEvaluationCb
bartlomieju Jan 13, 2024
0942640
It actually works as expected :O
bartlomieju Jan 13, 2024
608dc82
remove dead code
bartlomieju Jan 14, 2024
2fbfb41
more cleanup
bartlomieju Jan 14, 2024
ee0fe99
delete unneeded code
bartlomieju Jan 14, 2024
f41d3db
remove more dead code
bartlomieju Jan 14, 2024
1b0a001
Merge branch 'main' into request_module_type2
bartlomieju Jan 14, 2024
58b2c23
revert irrelevant changes
bartlomieju Jan 14, 2024
5a79df7
Merge branch 'main' into request_module_type2
bartlomieju Jan 14, 2024
a01d34b
Merge branch 'main' into request_module_type2
bartlomieju Jan 15, 2024
9b30224
Add CustomModuleEvaluationCtx
bartlomieju Jan 15, 2024
11c564a
wire it up all the way through
bartlomieju Jan 15, 2024
6117b45
Merge branch 'main' into request_module_type2
bartlomieju Jan 25, 2024
0b60dee
dedup
bartlomieju Jan 25, 2024
9fe242d
dedup2
bartlomieju Jan 25, 2024
7e28f0c
Merge branch 'main' into request_module_type2
bartlomieju Jan 31, 2024
6c62dba
refactor: make static strings use easier
bartlomieju Jan 31, 2024
46a4d8b
factor out more
bartlomieju Jan 31, 2024
d2ae26e
more cleanup
bartlomieju Jan 31, 2024
fd141f4
capture bindings
bartlomieju Jan 31, 2024
0ef2776
finish
bartlomieju Jan 31, 2024
f6aded2
Merge branch 'main' into wasm_analysis
bartlomieju Jan 31, 2024
0234b44
not available during snapshotting
bartlomieju Jan 31, 2024
41087ae
Merge branch 'wasm_analysis' into request_module_type2
bartlomieju Jan 31, 2024
8a76501
Merge branch 'main' into request_module_type2
bartlomieju Feb 1, 2024
55c6846
Merge branch 'main' into request_module_type2
bartlomieju Feb 1, 2024
e0562dd
use wasm_dep_analyzer
bartlomieju Feb 1, 2024
0e352cc
Merge branch 'main' into request_module_type2
bartlomieju Feb 2, 2024
efeb054
add TODO
bartlomieju Feb 2, 2024
423b7a1
store a callback in state
bartlomieju Feb 2, 2024
42ffd1c
remove CustomModuleEvaluationCtx
bartlomieju Feb 2, 2024
42a7942
Merge branch 'main' into request_module_type2
bartlomieju Feb 2, 2024
9247ade
wip
bartlomieju Feb 2, 2024
d60efcd
something is working
bartlomieju Feb 2, 2024
975985a
properly check for the module type
bartlomieju Feb 2, 2024
1a4d119
Merge branch 'main' into request_module_type2
bartlomieju Feb 2, 2024
84af4c8
remove debug output
bartlomieju Feb 2, 2024
818dc43
cleanup
bartlomieju Feb 2, 2024
b8e3ca5
add a test to render js wasm file
bartlomieju Feb 3, 2024
2ea9541
check for import.meta.wasmInstantiate
bartlomieju Feb 3, 2024
3009ad2
Merge branch 'main' into request_module_type2
bartlomieju Feb 5, 2024
6a329f8
revert unrelated changes
bartlomieju Feb 5, 2024
1312076
remove big file
bartlomieju Feb 5, 2024
63bcda5
move example to testing data
bartlomieju Feb 5, 2024
59b1dd9
add a test
bartlomieju Feb 5, 2024
94c4f7c
copyrights
bartlomieju Feb 5, 2024
70f556d
lint
bartlomieju Feb 5, 2024
0e39b08
regenerate .wasm using wasmer
bartlomieju Feb 5, 2024
b671da6
renames
bartlomieju Feb 5, 2024
04230cf
add instruction how to regenerate the file
bartlomieju Feb 5, 2024
8a034d7
remove wasmer dep
bartlomieju Feb 5, 2024
320a273
lint
bartlomieju Feb 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions core/examples/fs_module_loader.rs
Original file line number Diff line number Diff line change
@@ -1,12 +1,88 @@
// Copyright 2018-2023 the Deno authors. All rights reserved. MIT license.

use anyhow::anyhow;
use anyhow::Context;
use deno_core::anyhow::Error;
use deno_core::v8;
use deno_core::FastString;
use deno_core::FsModuleLoader;
use deno_core::JsRuntime;
use deno_core::ModuleSourceCode;
use deno_core::RuntimeOptions;
use std::borrow::Cow;
use std::collections::HashMap;
use std::rc::Rc;

fn custom_module_evaluation_cb(
scope: &mut v8::HandleScope,
module_type: Cow<'_, str>,
module_name: &FastString,
code: ModuleSourceCode,
) -> Result<v8::Global<v8::Value>, Error> {
match &*module_type {
"bytes" => Ok(bytes_module(scope, code)),
"text" => text_module(scope, module_name, code),
"url" => Ok(url_module(scope, module_name)),
bartlomieju marked this conversation as resolved.
Show resolved Hide resolved
_ => Err(anyhow!(
"Can't import {:?} because of unknown module type {}",
module_name,
module_type
)),
}
}

fn bytes_module(
scope: &mut v8::HandleScope,
code: ModuleSourceCode,
) -> v8::Global<v8::Value> {
// FsModuleLoader always returns bytes.
let ModuleSourceCode::Bytes(buf) = code else {
unreachable!()
};
let owned_buf = buf.to_vec();
let buf_len: usize = owned_buf.len();
let backing_store = v8::ArrayBuffer::new_backing_store_from_vec(owned_buf);
let backing_store_shared = backing_store.make_shared();
let ab = v8::ArrayBuffer::with_backing_store(scope, &backing_store_shared);
let uint8_array = v8::Uint8Array::new(scope, ab, 0, buf_len).unwrap();
let value: v8::Local<v8::Value> = uint8_array.into();
v8::Global::new(scope, value)
}

fn text_module(
scope: &mut v8::HandleScope,
module_name: &FastString,
code: ModuleSourceCode,
) -> Result<v8::Global<v8::Value>, Error> {
// FsModuleLoader always returns bytes.
let ModuleSourceCode::Bytes(buf) = code else {
unreachable!()
};

let code = std::str::from_utf8(buf.as_bytes()).with_context(|| {
format!("Can't convert {:?} source code to string", module_name)
})?;
let str_ = v8::String::new(scope, code).unwrap();
let value: v8::Local<v8::Value> = str_.into();
Ok(v8::Global::new(scope, value))
}

fn url_module(
scope: &mut v8::HandleScope,
module_name: &FastString,
) -> v8::Global<v8::Value> {
let str_ = v8::String::new(scope, module_name.as_str()).unwrap();
let value: v8::Local<v8::Value> = str_.into();
v8::Global::new(scope, value)
}

fn validate_import_attributes(
_scope: &mut v8::HandleScope,
_assertions: &HashMap<String, String>,
) {
// allow all
}

fn main() -> Result<(), Error> {
let args: Vec<String> = std::env::args().collect();
if args.len() < 2 {
Expand All @@ -18,6 +94,8 @@ fn main() -> Result<(), Error> {

let mut js_runtime = JsRuntime::new(RuntimeOptions {
module_loader: Some(Rc::new(FsModuleLoader)),
custom_module_evaluation_cb: Some(Box::new(custom_module_evaluation_cb)),
validate_import_attributes_cb: Some(Box::new(validate_import_attributes)),
..Default::default()
});

Expand Down
17 changes: 12 additions & 5 deletions core/modules/loaders.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ use crate::modules::ModuleCodeString;
use crate::modules::ModuleSource;
use crate::modules::ModuleSourceFuture;
use crate::modules::ModuleType;
use crate::modules::RequestedModuleType;
use crate::modules::ResolutionKind;
use crate::resolve_import;
use crate::Extension;
Expand All @@ -24,8 +25,6 @@ use std::future::Future;
use std::pin::Pin;
use std::rc::Rc;

use super::RequestedModuleType;

pub trait ModuleLoader {
/// Returns an absolute URL.
/// When implementing an spec-complaint VM, this should be exactly the
Expand Down Expand Up @@ -289,10 +288,11 @@ impl ModuleLoader for FsModuleLoader {
module_specifier: &ModuleSpecifier,
_maybe_referrer: Option<&ModuleSpecifier>,
_is_dynamic: bool,
_requested_module_type: RequestedModuleType,
requested_module_type: RequestedModuleType,
) -> Pin<Box<ModuleSourceFuture>> {
fn load(
module_specifier: &ModuleSpecifier,
requested_module_type: RequestedModuleType,
) -> Result<ModuleSource, AnyError> {
let path = module_specifier.to_file_path().map_err(|_| {
generic_error(format!(
Expand All @@ -301,10 +301,16 @@ impl ModuleLoader for FsModuleLoader {
})?;
let module_type = if let Some(extension) = path.extension() {
let ext = extension.to_string_lossy().to_lowercase();
// We only return JSON modules if extension was actually `.json`.
// In other cases we defer to actual requested module type, so runtime
// can decide what to do with it.
if ext == "json" {
ModuleType::Json
} else {
ModuleType::JavaScript
match requested_module_type {
RequestedModuleType::Other(ty) => ModuleType::Other(ty.clone()),
_ => ModuleType::JavaScript,
}
}
} else {
ModuleType::JavaScript
Expand All @@ -321,7 +327,8 @@ impl ModuleLoader for FsModuleLoader {
Ok(module)
}

futures::future::ready(load(module_specifier)).boxed_local()
futures::future::ready(load(module_specifier, requested_module_type))
.boxed_local()
}
}

Expand Down
90 changes: 64 additions & 26 deletions core/modules/map.rs
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,7 @@ impl ModuleMap {
return Ok(module_id);
}

let module_id = match module_type.clone() {
let module_id = match module_type {
ModuleType::JavaScript => {
let code =
ModuleSource::get_string_source(module_url_found.as_str(), code)
Expand All @@ -317,31 +317,69 @@ impl ModuleMap {
self.new_json_module(scope, module_url_found, code)?
}
ModuleType::Other(module_type) => {
let state = JsRuntime::state_from(scope);
let custom_module_evaluation_cb =
state.custom_module_evaluation_cb.as_ref();

let Some(custom_evaluation_cb) = custom_module_evaluation_cb else {
return Err(ModuleError::Other(generic_error(format!(
"Importing '{}' modules is not supported",
module_type
))));
};

let value_global = custom_evaluation_cb(
scope,
module_type.clone(),
&module_url_found,
code,
)
.map_err(ModuleError::Other)?;
let value = v8::Local::new(scope, value_global);
self.new_synthetic_module(
scope,
module_url_found,
ModuleType::Other(module_type.clone()),
value,
)?
// TODO(bartlomieju): this needs to be abstracted away and assumes
// "bytes" import support
if module_type == "wasm" {
let source = format!(
r#"
import wasmBytes from "{}" with {{ type: "bytes" }};
const wasmMod = await WebAssembly.compile(wasmBytes);
const requestedImports = WebAssembly.Module.imports(wasmMod);
const importedModules = await Promise.all(
requestedImports.map((i) => i.module).map((m) => import(m)),
);
const importsObject = {{}};
for (let i = 0; i < requestedImports.length; i++) {{
const importedModule = importedModules[i];
const requestedImport = requestedImports[i];
if (typeof importsObject[requestedImport.module] === "undefined") {{
importsObject[requestedImport.module] = {{}};
}}
const import_ = importedModule[requestedImport.name];
importsObject[requestedImport.module][requestedImport.name] = import_;
}}
const result = await WebAssembly.instantiate(wasmMod, importsObject);
export default result;
"#,
module_url_found.as_str()
);
// Ha! This is cool - we're saying this is a "WASM" module type, even
// though it's really a JS module that pulls WASM byte source.
self.new_module_from_js_source(
scope,
main,
ModuleType::Other("wasm".into()),
module_url_found,
source.into(),
dynamic,
)?
} else {
let state = JsRuntime::state_from(scope);
let custom_module_evaluation_cb =
state.custom_module_evaluation_cb.as_ref();

let Some(custom_evaluation_cb) = custom_module_evaluation_cb else {
return Err(ModuleError::Other(generic_error(format!(
"Importing '{}' modules is not supported",
module_type
))));
};

let value_global = custom_evaluation_cb(
scope,
module_type.clone(),
&module_url_found,
code,
)
.map_err(ModuleError::Other)?;
let value = v8::Local::new(scope, value_global);
self.new_synthetic_module(
scope,
module_url_found,
ModuleType::Other(module_type.clone()),
value,
)?
}
}
};
Ok(module_id)
Expand Down
Binary file added import_bytes/img.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
14 changes: 14 additions & 0 deletions import_bytes/main.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import img from "./img.jpg" with { type: "bytes" };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be an integration test

import imgUrl from "./img.jpg" with { type: "url" };
import textBytes from "./text.txt" with { type: "bytes" };
import text from "./text.txt" with { type: "text" };
import textUrl from "./text.txt" with { type: "url" };

console.log("Img byte length", img.length);
console.log("Img bytes", img.slice(0, 5));
console.log("Text bytes length", textBytes.length);
console.log("Text bytes", textBytes.slice(0, 5));
console.log("Text length", text.length);
console.log("Text", text.slice(0, 100));
console.log("Img url", imgUrl);
console.log("Text url", textUrl);
1 change: 1 addition & 0 deletions import_bytes/text.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
lorem ipsum dolor sit aemt
6 changes: 6 additions & 0 deletions wasm_example/import.mjs
bartlomieju marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import wasmBytes from "./import.wasm" with { type: "bytes" };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be an integration test

import { wasmShim } from "./test.js";

const wasmMod = await wasmShim(wasmBytes);
console.log(`hey ${wasmMod.exports.exported_add}`);
console.log(`${wasmMod.exports.exported_add()}`);
Binary file added wasm_example/import.wasm
Binary file not shown.
8 changes: 8 additions & 0 deletions wasm_example/import.wat
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
(module
(func $add (import "./import_inner.mjs" "add") (param i32) (param i32) (result i32))
(func (export "exported_add") (result i32)
i32.const 21
i32.const 21
call $add
)
)
14 changes: 14 additions & 0 deletions wasm_example/import2.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import wasmModule from "./import.wasm" with { type: "wasm" };
import bytes from "./import.wasm" with { type: "bytes" };
console.log(`hey ${exported_add}`);
console.log(`${exported_add()}`);

import {
get_plugin_schema_version,
get_wasm_memory_buffer_size,
} from "./plugin.wasm" with { type: "wasm" };
console.log("dprint plugin version", get_plugin_schema_version());
console.log(
"dprint plugin get_wasm_memory_buffer_size",
get_wasm_memory_buffer_size(),
);
14 changes: 14 additions & 0 deletions wasm_example/import3.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import wasmMod from "./import.wasm" with { type: "wasm" };
console.log(`hey ${wasmMod.exports.exported_add}`);
console.log(`${wasmMod.exports.exported_add()}`);

import dprintPlugin from "./plugin.wasm" with { type: "wasm" };
console.log(`hey ${Object.keys(dprintPlugin.exports)}`);
console.log(
"dprint plugin version",
dprintPlugin.exports.get_plugin_schema_version(),
);
console.log(
"dprint plugin get_wasm_memory_buffer_size",
dprintPlugin.exports.get_wasm_memory_buffer_size(),
);
8 changes: 8 additions & 0 deletions wasm_example/import_inner.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import { sleep } from "./import_inner_inner.mjs";
export { add } from "./import_inner_inner.mjs";

console.log("import_inner.js before");

await sleep(100);

console.log("import_inner.js after");
21 changes: 21 additions & 0 deletions wasm_example/import_inner_inner.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
console.log("import_inner_inner.js before");

export async function sleep(timeout) {
return new Promise((resolve) => {
Deno.core.queueTimer(
Deno.core.getTimerDepth() + 1,
false,
timeout,
resolve,
);
});
}
await sleep(100);

console.log("import_inner_inner.js after");

const abc = 1 + 2;
export function add(a, b) {
console.log(`abc: ${abc}`);
return a + b;
}
Binary file added wasm_example/plugin.wasm
Binary file not shown.
34 changes: 34 additions & 0 deletions wasm_example/test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
import wasmBytes from "./specifier.wasm" with { type: "bytes" };

export async function wasmShim(wasmBytes) {
const wasmMod = await WebAssembly.compile(wasmBytes);
const requestedImports = WebAssembly.Module.imports(wasmMod);
const importedModules = await Promise.all(
requestedImports.map((i) => i.module).map((m) => import(m)),
);
const importsObject = {};
for (let i = 0; i < requestedImports.length; i++) {
const importedModule = importedModules[i];
const requestedImport = requestedImports[i];
if (typeof importsObject[requestedImport.module] === "undefined") {
importsObject[requestedImport.module] = {};
}
const import_ = importedModule[requestedImport.name];
importsObject[requestedImport.module][requestedImport.name] = import_;
}
const result = await WebAssembly.instantiate(wasmMod, importsObject);
result.exports
result.imports;
return result;
}


///
const wasmMod = await WebAssembly.compile(wasmBytes);
const requestedImports = WebAssembly.Module.imports(wasmMod);
const exports = WebAssembly.Module.exports(wasmMod);

///

import import from "./"

Loading