-
Notifications
You must be signed in to change notification settings - Fork 94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Allow support for Wasm modules #402
Open
bartlomieju
wants to merge
82
commits into
denoland:main
Choose a base branch
from
bartlomieju:request_module_type2
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+320
−17
Open
Changes from 77 commits
Commits
Show all changes
82 commits
Select commit
Hold shift + click to select a range
4af042c
rename 'AssertedModuleType' to 'RequestedModuleType'
bartlomieju 394c674
more renames
bartlomieju 3404e94
test that creating a custom module fails
bartlomieju b0197b7
Merge branch 'main' into request_module_type2
bartlomieju 55eb3aa
custom module evaluation cb
bartlomieju f6106ed
wire up the callback
bartlomieju 3e88246
something is working, but module type is broken
bartlomieju 907f640
fmt
bartlomieju 4abc9f3
text import working too
bartlomieju 444d7b4
and url import work too
bartlomieju 333a934
a bit of cleanup
bartlomieju bbc40db
Divy's suggestion
bartlomieju 611401d
Add ModuleSourceBytes enum
bartlomieju b1dadd8
support top-level WASM imports
bartlomieju f4d9366
add dprint plugin
bartlomieju 498d018
Merge branch 'main' into request_module_type2
bartlomieju cedff3f
Merge branch 'main' into request_module_type2
bartlomieju 3d61bd9
this is how it should look like
bartlomieju 21bd425
keep so I don't forget
bartlomieju b81c0fd
Merge branch 'main' into request_module_type2
bartlomieju 216ec1f
Merge branch 'main' into request_module_type2
bartlomieju 28a0d97
dedup
bartlomieju 360271d
Merge branch 'main' into request_module_type2
bartlomieju 902fae4
fix after merge
bartlomieju c28fd3c
remove url imports
bartlomieju 8c98195
deny json imports if not json
bartlomieju 9573e6f
wasm-module import to show PoC of exported members of WASM module
bartlomieju ba2b85f
lint
bartlomieju cf9a119
some cleanup, add Deno.core.isWasmModuleObject()
bartlomieju 7e7535a
add CustomModuleEvaluationKind enum
bartlomieju 7dbc4b1
no ModuleMap in CustomModuleEvaluationCb
bartlomieju 0942640
It actually works as expected :O
bartlomieju 608dc82
remove dead code
bartlomieju 2fbfb41
more cleanup
bartlomieju ee0fe99
delete unneeded code
bartlomieju f41d3db
remove more dead code
bartlomieju 1b0a001
Merge branch 'main' into request_module_type2
bartlomieju 58b2c23
revert irrelevant changes
bartlomieju 5a79df7
Merge branch 'main' into request_module_type2
bartlomieju a01d34b
Merge branch 'main' into request_module_type2
bartlomieju 9b30224
Add CustomModuleEvaluationCtx
bartlomieju 11c564a
wire it up all the way through
bartlomieju 6117b45
Merge branch 'main' into request_module_type2
bartlomieju 0b60dee
dedup
bartlomieju 9fe242d
dedup2
bartlomieju 7e28f0c
Merge branch 'main' into request_module_type2
bartlomieju 6c62dba
refactor: make static strings use easier
bartlomieju 46a4d8b
factor out more
bartlomieju d2ae26e
more cleanup
bartlomieju fd141f4
capture bindings
bartlomieju 0ef2776
finish
bartlomieju f6aded2
Merge branch 'main' into wasm_analysis
bartlomieju 0234b44
not available during snapshotting
bartlomieju 41087ae
Merge branch 'wasm_analysis' into request_module_type2
bartlomieju 8a76501
Merge branch 'main' into request_module_type2
bartlomieju 55c6846
Merge branch 'main' into request_module_type2
bartlomieju e0562dd
use wasm_dep_analyzer
bartlomieju 0e352cc
Merge branch 'main' into request_module_type2
bartlomieju efeb054
add TODO
bartlomieju 423b7a1
store a callback in state
bartlomieju 42ffd1c
remove CustomModuleEvaluationCtx
bartlomieju 42a7942
Merge branch 'main' into request_module_type2
bartlomieju 9247ade
wip
bartlomieju d60efcd
something is working
bartlomieju 975985a
properly check for the module type
bartlomieju 1a4d119
Merge branch 'main' into request_module_type2
bartlomieju 84af4c8
remove debug output
bartlomieju 818dc43
cleanup
bartlomieju b8e3ca5
add a test to render js wasm file
bartlomieju 2ea9541
check for import.meta.wasmInstantiate
bartlomieju 3009ad2
Merge branch 'main' into request_module_type2
bartlomieju 6a329f8
revert unrelated changes
bartlomieju 1312076
remove big file
bartlomieju 63bcda5
move example to testing data
bartlomieju 59b1dd9
add a test
bartlomieju 94c4f7c
copyrights
bartlomieju 70f556d
lint
bartlomieju 0e39b08
regenerate .wasm using wasmer
bartlomieju b671da6
renames
bartlomieju 04230cf
add instruction how to regenerate the file
bartlomieju 8a034d7
remove wasmer dep
bartlomieju 320a273
lint
bartlomieju File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -23,8 +23,10 @@ use crate::runtime::JsRealm; | |
use crate::ExtensionFileSource; | ||
use crate::FastString; | ||
use crate::JsRuntime; | ||
use crate::ModuleCodeBytes; | ||
use crate::ModuleLoadResponse; | ||
use crate::ModuleSource; | ||
use crate::ModuleSourceCode; | ||
use crate::ModuleSpecifier; | ||
use anyhow::bail; | ||
use anyhow::Context as _; | ||
|
@@ -36,9 +38,11 @@ use futures::task::noop_waker_ref; | |
use futures::task::AtomicWaker; | ||
use futures::Future; | ||
use futures::StreamExt; | ||
use indexmap::IndexMap; | ||
use log::debug; | ||
use v8::Function; | ||
use v8::PromiseState; | ||
use wasm_dep_analyzer::WasmDeps; | ||
|
||
use std::cell::Cell; | ||
use std::cell::RefCell; | ||
|
@@ -298,7 +302,6 @@ impl ModuleMap { | |
); | ||
return Ok(module_id); | ||
} | ||
|
||
let module_id = match module_type { | ||
ModuleType::JavaScript => { | ||
let code = | ||
|
@@ -314,9 +317,12 @@ impl ModuleMap { | |
)? | ||
} | ||
ModuleType::Wasm => { | ||
return Err(ModuleError::Other(generic_error( | ||
"Importing Wasm modules is currently not supported.", | ||
))); | ||
let ModuleSourceCode::Bytes(code) = code else { | ||
return Err(ModuleError::Other(generic_error( | ||
"Source code for Wasm module must be provided as bytes", | ||
))); | ||
}; | ||
self.new_wasm_module(scope, module_url_found, code, dynamic)? | ||
} | ||
ModuleType::Json => { | ||
let code = | ||
|
@@ -598,6 +604,53 @@ impl ModuleMap { | |
Ok(id) | ||
} | ||
|
||
pub(crate) fn new_wasm_module( | ||
&self, | ||
scope: &mut v8::HandleScope, | ||
name: ModuleName, | ||
source: ModuleCodeBytes, | ||
is_dynamic_import: bool, | ||
) -> Result<ModuleId, ModuleError> { | ||
let bytes = source.as_bytes(); | ||
let wasm_module_analysis = WasmDeps::parse(bytes).map_err(|e| { | ||
let err = Error::from(e); | ||
ModuleError::Other(err) | ||
})?; | ||
|
||
let Some(wasm_module) = v8::WasmModuleObject::compile(scope, bytes) else { | ||
return Err(ModuleError::Other(generic_error(format!( | ||
"Failed to compile WASM module '{}'", | ||
name.as_str() | ||
)))); | ||
}; | ||
let wasm_module_value: v8::Local<v8::Value> = wasm_module.into(); | ||
|
||
let js_wasm_module_source = | ||
render_js_wasm_module(name.as_str(), wasm_module_analysis); | ||
|
||
let synthetic_module_type = | ||
ModuleType::Other("$$deno-core-internal-wasm-module".into()); | ||
|
||
let (name1, name2) = name.into_cheap_copy(); | ||
let value = v8::Local::new(scope, wasm_module_value); | ||
let exports = vec![(FastString::StaticAscii("default"), value)]; | ||
let _synthetic_mod_id = self.new_synthetic_module( | ||
scope, | ||
name1, | ||
synthetic_module_type, | ||
exports, | ||
)?; | ||
|
||
self.new_module_from_js_source( | ||
scope, | ||
false, | ||
ModuleType::Wasm, | ||
name2, | ||
js_wasm_module_source.into(), | ||
is_dynamic_import, | ||
) | ||
} | ||
|
||
pub(crate) fn new_json_module( | ||
&self, | ||
scope: &mut v8::HandleScope, | ||
|
@@ -1617,3 +1670,187 @@ pub fn module_origin<'a>( | |
true, | ||
) | ||
} | ||
|
||
fn render_js_wasm_module(specifier: &str, wasm_deps: WasmDeps) -> String { | ||
// NOTE(bartlomieju): it's unlikely the generated file will have more lines, | ||
// but it's better to overallocate than to have to mem copy. | ||
let mut src = Vec::with_capacity(512); | ||
|
||
fn aggregate_wasm_module_imports( | ||
imports: &[wasm_dep_analyzer::Import], | ||
) -> IndexMap<String, Vec<String>> { | ||
let mut imports_map = IndexMap::default(); | ||
|
||
for import in imports.iter().filter(|i| { | ||
matches!(i.import_type, wasm_dep_analyzer::ImportType::Function(..)) | ||
}) { | ||
let entry = imports_map | ||
.entry(import.module.to_string()) | ||
.or_insert(vec![]); | ||
entry.push(import.name.to_string()); | ||
} | ||
|
||
imports_map | ||
} | ||
|
||
src.push(format!( | ||
r#"import wasmMod from "{}" with {{ type: "$$deno-core-internal-wasm-module" }};"#, | ||
specifier, | ||
)); | ||
|
||
// TODO(bartlomieju): handle imports collisions? | ||
if !wasm_deps.imports.is_empty() { | ||
let aggregated_imports = aggregate_wasm_module_imports(&wasm_deps.imports); | ||
|
||
for (key, value) in aggregated_imports.iter() { | ||
src.push(format!( | ||
r#"import {{ {} }} from "{}";"#, | ||
value.join(", "), | ||
key | ||
)); | ||
} | ||
|
||
src.push("const importsObject = {".to_string()); | ||
|
||
for (key, value) in aggregated_imports.iter() { | ||
src.push(format!(" \"{}\": {{", key).to_string()); | ||
|
||
for el in value { | ||
src.push(format!(" {},", el)); | ||
} | ||
|
||
src.push(" },".to_string()); | ||
} | ||
|
||
src.push("};".to_string()); | ||
|
||
src.push( | ||
"const modInstance = await import.meta.wasmInstantiate(wasmMod, importsObject);".to_string(), | ||
) | ||
} else { | ||
src.push( | ||
"const modInstance = await import.meta.wasmInstantiate(wasmMod);" | ||
.to_string(), | ||
) | ||
} | ||
|
||
if !wasm_deps.exports.is_empty() { | ||
for export_desc in wasm_deps.exports.iter().filter(|e| { | ||
matches!(e.export_type, wasm_dep_analyzer::ExportType::Function) | ||
}) { | ||
if export_desc.name == "default" { | ||
src.push(format!( | ||
"export default modInstance.exports.{};", | ||
export_desc.name | ||
)); | ||
} else { | ||
src.push(format!( | ||
"export const {} = modInstance.exports.{};", | ||
export_desc.name, export_desc.name | ||
)); | ||
} | ||
} | ||
} | ||
|
||
src.join("\n") | ||
} | ||
|
||
#[test] | ||
fn test_render_js_wasm_module() { | ||
let deps = WasmDeps { | ||
imports: vec![], | ||
exports: vec![], | ||
}; | ||
let rendered = render_js_wasm_module("./foo.wasm", deps); | ||
pretty_assertions::assert_eq!( | ||
rendered, | ||
r#"import wasmMod from "./foo.wasm" with { type: "$$deno-core-internal-wasm-module" }; | ||
const modInstance = await import.meta.wasmInstantiate(wasmMod);"#, | ||
); | ||
|
||
let deps = WasmDeps { | ||
imports: vec![ | ||
wasm_dep_analyzer::Import { | ||
name: "foo", | ||
module: "./import.js", | ||
import_type: wasm_dep_analyzer::ImportType::Tag( | ||
wasm_dep_analyzer::TagType { | ||
kind: 1, | ||
type_index: 1, | ||
}, | ||
), | ||
}, | ||
wasm_dep_analyzer::Import { | ||
name: "bar", | ||
module: "./import.js", | ||
import_type: wasm_dep_analyzer::ImportType::Function(1), | ||
}, | ||
wasm_dep_analyzer::Import { | ||
name: "fizz", | ||
module: "./import.js", | ||
import_type: wasm_dep_analyzer::ImportType::Function(2), | ||
}, | ||
wasm_dep_analyzer::Import { | ||
name: "buzz", | ||
module: "./buzz.js", | ||
import_type: wasm_dep_analyzer::ImportType::Function(3), | ||
}, | ||
], | ||
exports: vec![ | ||
wasm_dep_analyzer::Export { | ||
name: "export1", | ||
index: 0, | ||
export_type: wasm_dep_analyzer::ExportType::Function, | ||
}, | ||
wasm_dep_analyzer::Export { | ||
name: "export2", | ||
index: 1, | ||
export_type: wasm_dep_analyzer::ExportType::Table, | ||
}, | ||
wasm_dep_analyzer::Export { | ||
name: "export3", | ||
index: 2, | ||
export_type: wasm_dep_analyzer::ExportType::Memory, | ||
}, | ||
wasm_dep_analyzer::Export { | ||
name: "export4", | ||
index: 3, | ||
export_type: wasm_dep_analyzer::ExportType::Global, | ||
}, | ||
wasm_dep_analyzer::Export { | ||
name: "export5", | ||
index: 4, | ||
export_type: wasm_dep_analyzer::ExportType::Tag, | ||
}, | ||
wasm_dep_analyzer::Export { | ||
name: "export6", | ||
index: 5, | ||
export_type: wasm_dep_analyzer::ExportType::Unknown, | ||
}, | ||
wasm_dep_analyzer::Export { | ||
name: "default", | ||
index: 6, | ||
export_type: wasm_dep_analyzer::ExportType::Function, | ||
}, | ||
], | ||
}; | ||
let rendered = render_js_wasm_module("./foo.wasm", deps); | ||
pretty_assertions::assert_eq!( | ||
rendered, | ||
r#"import wasmMod from "./foo.wasm" with { type: "$$deno-core-internal-wasm-module" }; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Store as a .out file? |
||
import { bar, fizz } from "./import.js"; | ||
import { buzz } from "./buzz.js"; | ||
const importsObject = { | ||
"./import.js": { | ||
bar, | ||
fizz, | ||
}, | ||
"./buzz.js": { | ||
buzz, | ||
}, | ||
}; | ||
const modInstance = await import.meta.wasmInstantiate(wasmMod, importsObject); | ||
export const export1 = modInstance.exports.export1; | ||
export default modInstance.exports.default;"#, | ||
); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could probably store some basic WASM modules in the testdata and simplify these tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WasmDeps
doesn't implementSerialize
/Deserialize
so there's no easy way to store intermediate representation. I agree that it could be improved, but maybe it's not a blocker for this PR?