Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow support for Wasm modules #402

Open
wants to merge 82 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 77 commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
4af042c
rename 'AssertedModuleType' to 'RequestedModuleType'
bartlomieju Dec 29, 2023
394c674
more renames
bartlomieju Dec 29, 2023
3404e94
test that creating a custom module fails
bartlomieju Dec 29, 2023
b0197b7
Merge branch 'main' into request_module_type2
bartlomieju Dec 29, 2023
55eb3aa
custom module evaluation cb
bartlomieju Dec 29, 2023
f6106ed
wire up the callback
bartlomieju Dec 29, 2023
3e88246
something is working, but module type is broken
bartlomieju Dec 29, 2023
907f640
fmt
bartlomieju Dec 29, 2023
4abc9f3
text import working too
bartlomieju Dec 29, 2023
444d7b4
and url import work too
bartlomieju Dec 29, 2023
333a934
a bit of cleanup
bartlomieju Dec 29, 2023
bbc40db
Divy's suggestion
bartlomieju Jan 1, 2024
611401d
Add ModuleSourceBytes enum
bartlomieju Jan 3, 2024
b1dadd8
support top-level WASM imports
bartlomieju Jan 3, 2024
f4d9366
add dprint plugin
bartlomieju Jan 3, 2024
498d018
Merge branch 'main' into request_module_type2
bartlomieju Jan 9, 2024
cedff3f
Merge branch 'main' into request_module_type2
bartlomieju Jan 11, 2024
3d61bd9
this is how it should look like
bartlomieju Jan 11, 2024
21bd425
keep so I don't forget
bartlomieju Jan 11, 2024
b81c0fd
Merge branch 'main' into request_module_type2
bartlomieju Jan 11, 2024
216ec1f
Merge branch 'main' into request_module_type2
bartlomieju Jan 11, 2024
28a0d97
dedup
bartlomieju Jan 11, 2024
360271d
Merge branch 'main' into request_module_type2
bartlomieju Jan 12, 2024
902fae4
fix after merge
bartlomieju Jan 12, 2024
c28fd3c
remove url imports
bartlomieju Jan 12, 2024
8c98195
deny json imports if not json
bartlomieju Jan 12, 2024
9573e6f
wasm-module import to show PoC of exported members of WASM module
bartlomieju Jan 13, 2024
ba2b85f
lint
bartlomieju Jan 13, 2024
cf9a119
some cleanup, add Deno.core.isWasmModuleObject()
bartlomieju Jan 13, 2024
7e7535a
add CustomModuleEvaluationKind enum
bartlomieju Jan 13, 2024
7dbc4b1
no ModuleMap in CustomModuleEvaluationCb
bartlomieju Jan 13, 2024
0942640
It actually works as expected :O
bartlomieju Jan 13, 2024
608dc82
remove dead code
bartlomieju Jan 14, 2024
2fbfb41
more cleanup
bartlomieju Jan 14, 2024
ee0fe99
delete unneeded code
bartlomieju Jan 14, 2024
f41d3db
remove more dead code
bartlomieju Jan 14, 2024
1b0a001
Merge branch 'main' into request_module_type2
bartlomieju Jan 14, 2024
58b2c23
revert irrelevant changes
bartlomieju Jan 14, 2024
5a79df7
Merge branch 'main' into request_module_type2
bartlomieju Jan 14, 2024
a01d34b
Merge branch 'main' into request_module_type2
bartlomieju Jan 15, 2024
9b30224
Add CustomModuleEvaluationCtx
bartlomieju Jan 15, 2024
11c564a
wire it up all the way through
bartlomieju Jan 15, 2024
6117b45
Merge branch 'main' into request_module_type2
bartlomieju Jan 25, 2024
0b60dee
dedup
bartlomieju Jan 25, 2024
9fe242d
dedup2
bartlomieju Jan 25, 2024
7e28f0c
Merge branch 'main' into request_module_type2
bartlomieju Jan 31, 2024
6c62dba
refactor: make static strings use easier
bartlomieju Jan 31, 2024
46a4d8b
factor out more
bartlomieju Jan 31, 2024
d2ae26e
more cleanup
bartlomieju Jan 31, 2024
fd141f4
capture bindings
bartlomieju Jan 31, 2024
0ef2776
finish
bartlomieju Jan 31, 2024
f6aded2
Merge branch 'main' into wasm_analysis
bartlomieju Jan 31, 2024
0234b44
not available during snapshotting
bartlomieju Jan 31, 2024
41087ae
Merge branch 'wasm_analysis' into request_module_type2
bartlomieju Jan 31, 2024
8a76501
Merge branch 'main' into request_module_type2
bartlomieju Feb 1, 2024
55c6846
Merge branch 'main' into request_module_type2
bartlomieju Feb 1, 2024
e0562dd
use wasm_dep_analyzer
bartlomieju Feb 1, 2024
0e352cc
Merge branch 'main' into request_module_type2
bartlomieju Feb 2, 2024
efeb054
add TODO
bartlomieju Feb 2, 2024
423b7a1
store a callback in state
bartlomieju Feb 2, 2024
42ffd1c
remove CustomModuleEvaluationCtx
bartlomieju Feb 2, 2024
42a7942
Merge branch 'main' into request_module_type2
bartlomieju Feb 2, 2024
9247ade
wip
bartlomieju Feb 2, 2024
d60efcd
something is working
bartlomieju Feb 2, 2024
975985a
properly check for the module type
bartlomieju Feb 2, 2024
1a4d119
Merge branch 'main' into request_module_type2
bartlomieju Feb 2, 2024
84af4c8
remove debug output
bartlomieju Feb 2, 2024
818dc43
cleanup
bartlomieju Feb 2, 2024
b8e3ca5
add a test to render js wasm file
bartlomieju Feb 3, 2024
2ea9541
check for import.meta.wasmInstantiate
bartlomieju Feb 3, 2024
3009ad2
Merge branch 'main' into request_module_type2
bartlomieju Feb 5, 2024
6a329f8
revert unrelated changes
bartlomieju Feb 5, 2024
1312076
remove big file
bartlomieju Feb 5, 2024
63bcda5
move example to testing data
bartlomieju Feb 5, 2024
59b1dd9
add a test
bartlomieju Feb 5, 2024
94c4f7c
copyrights
bartlomieju Feb 5, 2024
70f556d
lint
bartlomieju Feb 5, 2024
0e39b08
regenerate .wasm using wasmer
bartlomieju Feb 5, 2024
b671da6
renames
bartlomieju Feb 5, 2024
04230cf
add instruction how to regenerate the file
bartlomieju Feb 5, 2024
8a034d7
remove wasmer dep
bartlomieju Feb 5, 2024
320a273
lint
bartlomieju Feb 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ deno_core_icudata = { workspace = true, optional = true }
deno_ops.workspace = true
deno_unsync.workspace = true
futures.workspace = true
indexmap = "2.1.0"
libc.workspace = true
log.workspace = true
memoffset.workspace = true
Expand All @@ -49,6 +50,7 @@ static_assertions.workspace = true
tokio.workspace = true
url.workspace = true
v8.workspace = true
wasm_dep_analyzer = "0.0.1"

[dev-dependencies]
bencher.workspace = true
Expand Down
2 changes: 2 additions & 0 deletions core/modules/loaders.rs
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,8 @@ impl ModuleLoader for FsModuleLoader {
// can decide what to do with it.
if ext == "json" {
ModuleType::Json
} else if ext == "wasm" {
ModuleType::Wasm
} else {
match &requested_module_type {
RequestedModuleType::Other(ty) => ModuleType::Other(ty.clone()),
Expand Down
245 changes: 241 additions & 4 deletions core/modules/map.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,10 @@ use crate::runtime::JsRealm;
use crate::ExtensionFileSource;
use crate::FastString;
use crate::JsRuntime;
use crate::ModuleCodeBytes;
use crate::ModuleLoadResponse;
use crate::ModuleSource;
use crate::ModuleSourceCode;
use crate::ModuleSpecifier;
use anyhow::bail;
use anyhow::Context as _;
Expand All @@ -36,9 +38,11 @@ use futures::task::noop_waker_ref;
use futures::task::AtomicWaker;
use futures::Future;
use futures::StreamExt;
use indexmap::IndexMap;
use log::debug;
use v8::Function;
use v8::PromiseState;
use wasm_dep_analyzer::WasmDeps;

use std::cell::Cell;
use std::cell::RefCell;
Expand Down Expand Up @@ -298,7 +302,6 @@ impl ModuleMap {
);
return Ok(module_id);
}

let module_id = match module_type {
ModuleType::JavaScript => {
let code =
Expand All @@ -314,9 +317,12 @@ impl ModuleMap {
)?
}
ModuleType::Wasm => {
return Err(ModuleError::Other(generic_error(
"Importing Wasm modules is currently not supported.",
)));
let ModuleSourceCode::Bytes(code) = code else {
return Err(ModuleError::Other(generic_error(
"Source code for Wasm module must be provided as bytes",
)));
};
self.new_wasm_module(scope, module_url_found, code, dynamic)?
}
ModuleType::Json => {
let code =
Expand Down Expand Up @@ -598,6 +604,53 @@ impl ModuleMap {
Ok(id)
}

pub(crate) fn new_wasm_module(
&self,
scope: &mut v8::HandleScope,
name: ModuleName,
source: ModuleCodeBytes,
is_dynamic_import: bool,
) -> Result<ModuleId, ModuleError> {
let bytes = source.as_bytes();
let wasm_module_analysis = WasmDeps::parse(bytes).map_err(|e| {
let err = Error::from(e);
ModuleError::Other(err)
})?;

let Some(wasm_module) = v8::WasmModuleObject::compile(scope, bytes) else {
return Err(ModuleError::Other(generic_error(format!(
"Failed to compile WASM module '{}'",
name.as_str()
))));
};
let wasm_module_value: v8::Local<v8::Value> = wasm_module.into();

let js_wasm_module_source =
render_js_wasm_module(name.as_str(), wasm_module_analysis);

let synthetic_module_type =
ModuleType::Other("$$deno-core-internal-wasm-module".into());

let (name1, name2) = name.into_cheap_copy();
let value = v8::Local::new(scope, wasm_module_value);
let exports = vec![(FastString::StaticAscii("default"), value)];
let _synthetic_mod_id = self.new_synthetic_module(
scope,
name1,
synthetic_module_type,
exports,
)?;

self.new_module_from_js_source(
scope,
false,
ModuleType::Wasm,
name2,
js_wasm_module_source.into(),
is_dynamic_import,
)
}

pub(crate) fn new_json_module(
&self,
scope: &mut v8::HandleScope,
Expand Down Expand Up @@ -1617,3 +1670,187 @@ pub fn module_origin<'a>(
true,
)
}

fn render_js_wasm_module(specifier: &str, wasm_deps: WasmDeps) -> String {
// NOTE(bartlomieju): it's unlikely the generated file will have more lines,
// but it's better to overallocate than to have to mem copy.
let mut src = Vec::with_capacity(512);

fn aggregate_wasm_module_imports(
imports: &[wasm_dep_analyzer::Import],
) -> IndexMap<String, Vec<String>> {
let mut imports_map = IndexMap::default();

for import in imports.iter().filter(|i| {
matches!(i.import_type, wasm_dep_analyzer::ImportType::Function(..))
}) {
let entry = imports_map
.entry(import.module.to_string())
.or_insert(vec![]);
entry.push(import.name.to_string());
}

imports_map
}

src.push(format!(
r#"import wasmMod from "{}" with {{ type: "$$deno-core-internal-wasm-module" }};"#,
specifier,
));

// TODO(bartlomieju): handle imports collisions?
if !wasm_deps.imports.is_empty() {
let aggregated_imports = aggregate_wasm_module_imports(&wasm_deps.imports);

for (key, value) in aggregated_imports.iter() {
src.push(format!(
r#"import {{ {} }} from "{}";"#,
value.join(", "),
key
));
}

src.push("const importsObject = {".to_string());

for (key, value) in aggregated_imports.iter() {
src.push(format!(" \"{}\": {{", key).to_string());

for el in value {
src.push(format!(" {},", el));
}

src.push(" },".to_string());
}

src.push("};".to_string());

src.push(
"const modInstance = await import.meta.wasmInstantiate(wasmMod, importsObject);".to_string(),
)
} else {
src.push(
"const modInstance = await import.meta.wasmInstantiate(wasmMod);"
.to_string(),
)
}

if !wasm_deps.exports.is_empty() {
for export_desc in wasm_deps.exports.iter().filter(|e| {
matches!(e.export_type, wasm_dep_analyzer::ExportType::Function)
}) {
if export_desc.name == "default" {
src.push(format!(
"export default modInstance.exports.{};",
export_desc.name
));
} else {
src.push(format!(
"export const {} = modInstance.exports.{};",
export_desc.name, export_desc.name
));
}
}
}

src.join("\n")
}

#[test]
fn test_render_js_wasm_module() {
let deps = WasmDeps {
imports: vec![],
exports: vec![],
};
let rendered = render_js_wasm_module("./foo.wasm", deps);
pretty_assertions::assert_eq!(
rendered,
r#"import wasmMod from "./foo.wasm" with { type: "$$deno-core-internal-wasm-module" };
const modInstance = await import.meta.wasmInstantiate(wasmMod);"#,
);

let deps = WasmDeps {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could probably store some basic WASM modules in the testdata and simplify these tests.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WasmDeps doesn't implement Serialize/Deserialize so there's no easy way to store intermediate representation. I agree that it could be improved, but maybe it's not a blocker for this PR?

imports: vec![
wasm_dep_analyzer::Import {
name: "foo",
module: "./import.js",
import_type: wasm_dep_analyzer::ImportType::Tag(
wasm_dep_analyzer::TagType {
kind: 1,
type_index: 1,
},
),
},
wasm_dep_analyzer::Import {
name: "bar",
module: "./import.js",
import_type: wasm_dep_analyzer::ImportType::Function(1),
},
wasm_dep_analyzer::Import {
name: "fizz",
module: "./import.js",
import_type: wasm_dep_analyzer::ImportType::Function(2),
},
wasm_dep_analyzer::Import {
name: "buzz",
module: "./buzz.js",
import_type: wasm_dep_analyzer::ImportType::Function(3),
},
],
exports: vec![
wasm_dep_analyzer::Export {
name: "export1",
index: 0,
export_type: wasm_dep_analyzer::ExportType::Function,
},
wasm_dep_analyzer::Export {
name: "export2",
index: 1,
export_type: wasm_dep_analyzer::ExportType::Table,
},
wasm_dep_analyzer::Export {
name: "export3",
index: 2,
export_type: wasm_dep_analyzer::ExportType::Memory,
},
wasm_dep_analyzer::Export {
name: "export4",
index: 3,
export_type: wasm_dep_analyzer::ExportType::Global,
},
wasm_dep_analyzer::Export {
name: "export5",
index: 4,
export_type: wasm_dep_analyzer::ExportType::Tag,
},
wasm_dep_analyzer::Export {
name: "export6",
index: 5,
export_type: wasm_dep_analyzer::ExportType::Unknown,
},
wasm_dep_analyzer::Export {
name: "default",
index: 6,
export_type: wasm_dep_analyzer::ExportType::Function,
},
],
};
let rendered = render_js_wasm_module("./foo.wasm", deps);
pretty_assertions::assert_eq!(
rendered,
r#"import wasmMod from "./foo.wasm" with { type: "$$deno-core-internal-wasm-module" };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Store as a .out file?

import { bar, fizz } from "./import.js";
import { buzz } from "./buzz.js";
const importsObject = {
"./import.js": {
bar,
fizz,
},
"./buzz.js": {
buzz,
},
};
const modInstance = await import.meta.wasmInstantiate(wasmMod, importsObject);
export const export1 = modInstance.exports.export1;
export default modInstance.exports.default;"#,
);
}
1 change: 1 addition & 0 deletions core/modules/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ if (b() != 'b') throw Error();
if (c() != 'c') throw Error();
if (!import.meta.main) throw Error();
if (import.meta.url != 'file:///a.js') throw Error();
if (import.meta.wasmInstantiate !== undefined) throw Error();
"#;

const B_SRC: &str = r#"
Expand Down
2 changes: 1 addition & 1 deletion testing/checkin/runner/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ fn get_test_url(test_dir: &Path, test: &str) -> Result<Url, Error> {
}
}
let Some(path) = path else {
bail!("Test file not found");
bail!("Test file not found: {}.[ts.js.nocompile]", test);
};
let path = path.canonicalize()?.to_owned();
let url = Url::from_file_path(path).unwrap().to_string();
Expand Down
Loading
Loading