Loader: support .wasm imports #3328

kevinkassimo · 2019-11-13T07:16:04Z

import { myFunc } from "./my.wasm";
myFunc(10);

Imported WASM file can import from other JS files as follows

(module
  (import "./wasm-dep.js" "jsFn" (func $jsFn (result i32)))
  (import "./wasm-dep.js" "jsInitFn" (func $jsInitFn))
  (import "http://localhost:4545/cli/tests/051_wasm_import/remote.ts" "jsRemoteFn" (func $jsRemoteFn (result i32)))
  ;; ...
)

See cli/compilers/wasm.rs and cli/compilers/wasm_wrap.js for details.

Currently supporting only default exports since we currently have no access to Worker for running code from Compiler trait. (since also means currently we need --allow-read to do dynamic imports (doable under top-level await)). This is different from Node.js, which implements named exports, since Node tries to compile WASM module first (as it can run JS code before module creation), extracting wasmModule.imports and wasmModule.exports, and build a dynamic script with static imports and explicit named exports and compile the script as a separate ES module. See https://github.com/nodejs/node/blob/35ec01097b2a397ad0a22aac536fe07514876e21/lib/internal/modules/esm/translators.js#L190-L210

Named exports is implemented.

Usage of base64 encoding is debatable if we can accept failure of resolution given cases where Deno namespace is hidden. Braces in wasm_wrap.js are double braces to avoid breaking format!() macro. Using string replacements

kevinkassimo · 2019-11-13T07:32:23Z

Looks like test lock_check_err2 is badly flaky.

kevinkassimo · 2019-11-13T19:24:00Z

This actually also implies that ./target/debug/deno --allow-read my.wasm would also work.

kitsonk

Awseome 🎉 Just a couple of thoughts.

kitsonk · 2019-11-13T22:24:38Z

cli/js/compiler.ts

+        sourceFileJson.filename
+      );
+      // Create declaration file on the fly.
+      const _ = new SourceFile({


Why even assign this to a variable? It is never read.

But good way of tackling the problem. The interesting thing will be if we can actually get the shape of the named exports of the WASM file, and actually pass into the compiler a dynamically generated set of source code.

kitsonk · 2019-11-13T22:28:46Z

cli/tests/051_wasm_import.ts

@@ -0,0 +1,24 @@
+import wasmExports from "./051_wasm_import/simple.wasm";


Hmmm, is this the proposed semantics for WASM imports, for all of the exports to go into the default export, instead of import * as wasmExports from "..."?

The standards seems to imply named exports. However we have problem collecting WASM imports and exports names since our current compiler does not support running some JS code. I don’t really know if there is a way I can directly get these info from Rust side other than either delegate this task to JS or using something like wasmer_runtime (which is too heavy ad it is a full runtime). Will investigate if there is some V8 APIs I could use instead...

Actually I think I might be able to figure out export names with some extra tricks. Experimenting

I mean there are V8 APIs for WASM, which we would hook into at a lower-level...

https://github.com/v8/v8/blob/3c8c1c9d0a7d51cb8634dba2d829bd7d99d99163/include/v8.h#L1562-L1572
https://github.com/v8/v8/blob/3c8c1c9d0a7d51cb8634dba2d829bd7d99d99163/include/v8.h#L4648-L4668
https://github.com/v8/v8/blob/3c8c1c9d0a7d51cb8634dba2d829bd7d99d99163/include/v8.h#L4670-L4729

These APIs will be easier to access once we replace libdeno with rusty_v8

@ry I'm actually planning to do something similar to TsCompiler to delegate discovery of imports/exports through a separate worker, and construct a script based on the message posted back. These V8 APIs seems lacking access to import/export details...

I'm not sure what you need exactly but I'm sure it's there somewhere. Check out GetModuleNamespace.

@ry I believe that is from v8::internal that is not public?
Anyways I am now able to get named exports working now by having a JS worker compile and get imports/exports for me before I construct the static scripts.

It's here https://github.com/v8/v8/blob/3c8c1c9d0a7d51cb8634dba2d829bd7d99d99163/include/v8.h#L1542

ry · 2019-11-14T03:21:32Z

cli/compilers/wasm_wrap.js

@@ -0,0 +1,25 @@
+function base64ToUint8Array(data) {{


If this file is to have a .js filename extension, then it should be syntactically valid JS.

Rather than using format!, which induces these double braces, it would be better to use a template of some sort. Maybe you can just do a substring replacement on BASE64_ENCODED_WASM (or something)

ry · 2019-11-14T03:23:02Z

cli/compilers/wasm_wrap.js

+
+const instance = new WebAssembly.Instance(compiled, importObject);
+
+export default instance.exports;


What's cool about this is I think it would even work with deno bundle

yeah, actually that would simplify what I was going to do, there are consequences which I will detail in the related issue though.

kevinkassimo · 2019-11-14T06:35:32Z

Now supports named exports. Done through booting up a JS worker to extract imports/exports for us, sending back the info, and construct a static script based on it.

Bonus point: now imports are all static, so --allow-read flag can be dropped

ry

LGTM - very nice - I'm sure we'll need to iterate on this but I'm happy to land now that you've got the exports working.

* loader: support .wasm imports * http_server: true * Support named exports * Clippy

loader: support .wasm imports

22e4670

kevinkassimo force-pushed the loader/wasm branch from 48b64fd to 22e4670 Compare November 13, 2019 07:31

Merge branch 'master' into loader/wasm

a8cd4ab

kevinkassimo force-pushed the loader/wasm branch from ee3fdfb to a8cd4ab Compare November 13, 2019 19:13

http_server: true

e9a2763

kevinkassimo force-pushed the loader/wasm branch from bbad50e to e9a2763 Compare November 13, 2019 19:34

kitsonk reviewed Nov 13, 2019

View reviewed changes

ry reviewed Nov 14, 2019

View reviewed changes

Support named exports

aab0857

Clippy

ba35707

ry approved these changes Nov 14, 2019

View reviewed changes

Merge branch 'master' into loader/wasm

edd8ca4

ry merged commit 4189cc1 into denoland:master Nov 14, 2019

This was referenced Nov 14, 2019

Support importing wasm modules #2552

Closed

wasm import: remove double compilation #3348

Closed

bartlomieju pushed a commit to bartlomieju/deno that referenced this pull request Dec 28, 2019

Loader: support .wasm imports (denoland#3328)

daf4d56

* loader: support .wasm imports * http_server: true * Support named exports * Clippy

		@@ -0,0 +1,24 @@
		import wasmExports from "./051_wasm_import/simple.wasm";


		const instance = new WebAssembly.Instance(compiled, importObject);

		export default instance.exports;

Loader: support .wasm imports #3328

Loader: support .wasm imports #3328

Uh oh!

Conversation

kevinkassimo commented Nov 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinkassimo commented Nov 13, 2019

Uh oh!

kevinkassimo commented Nov 13, 2019

Uh oh!

kitsonk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ry Nov 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinkassimo Nov 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ry Nov 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinkassimo commented Nov 14, 2019

Uh oh!

ry left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kevinkassimo commented Nov 13, 2019 •

edited

Loading

ry Nov 14, 2019 •

edited

Loading

kevinkassimo Nov 14, 2019 •

edited

Loading

ry Nov 14, 2019 •

edited

Loading