Deno corrupts WASM binaries downloaded via network

[dyedgreen]

Deno allows to import WASM files directly as per #2552. This works great for local files. However, if the files are downloaded via the network, they seem to be corrupted, resulting in compile errors.

How to reproduce

(tested on deno 0.26.0)

1. compile add.c clang --target=wasm32 --no-standard-libraries -Wl,--no-entry -nostartfiles -Wl,--export-all -o add.wasm add.c
2. run server to host wasm file deno --allow-read --allow-net serve_wasm.js
3. try running the other js scripts deno import_xxx.js

Files

add.c

int add(int a, int b) {
  return a + b;
}

serve_wasm.js

import { serve } from "https://deno.land/std/http/server.ts";

const s = serve({ port: 8000 });

for await (const req of s) {
  console.log(req.url);
  req.respond({ body: await Deno.readFile("add.wasm") });
}

import_local.js

import * as mod from "./add.wasm";
// Works!
console.log(mod);

import_net.js

import * as mod from "http://localhost:8000/add.wasm";
// Fails with compile error
console.log(mod);

import_cache.js

import * as mod from "./deno_dir/deps/http/localhost_PORT8000/add.wasm";
// Also fails with compile error
console.log(mod);

Some observations

Comparing the served files with the file in the cache there are some discrepancies:

# Original file
334B Dec 20 13:19 add.wasm
# File in deno cache
356B Dec 20 13:24 add.wasm

And cating the files reveals that the cached version has a bunch of extra bytes:
original

asm
``p!A
     A
     A
     A
      A
      ^memory__wasm_call_ctors
__global_base
             __heap_base
                        __dso_handleadd
B
 =#!A!  k!  6
                (
                 ! !  j!
                          name__wasm_call_ctorsadd>     producers
                                                                 processed-byclang9.0.0 (tags/RELEASE_900/final)

deno cache

asm
``p!A��
       A�
        A�
         A��
            A�
             ^memory__wasm_call_ctors
__global_base
             __heap_base
                        __dso_handleadd
B
 =#����!A!  k!  6
                    (
                     ! !  j!
                              name__wasm_call_ctorsadd> producers
                                                                 processed-byclang9.0.0 (tags/RELEASE_900/final)

[dyedgreen]

Going through the source, I think the issue is as follows:

Deno UTF8-encodes the source code downloaded. However, doing this to the WASM binary corrupts it in the same way observed above. To fix this, WASM module downloads would need to bypass the UTF8-encoding step.

Example:

const bytes = await Deno.readFile("add.wasm");
const utf8 = new TextDecoder().decode(bytes);
const bytes_out = new TextEncoder().encode(utf8);

console.log(bytes.length, bytes_out.length);
console.log(utf8);

outputs:

334 356
asm
``p!A��
       A�
        A�
         A��
            A�
             ^memory__wasm_call_ctors
__global_base
             __heap_base
                        __dso_handleadd
B
 =#����!A!  k!  6
                    (
                     ! !  j!
                              name__wasm_call_ctorsadd> producers
                                                                 processed-byclang9.0.0 (tags/RELEASE_900/final)

[kevinkassimo]

I wonder this is a problem of our file_fetcher that is shared among all loaders, you'll notice code like http_util::fetch_string_once(...) which would possibly return a String -- which is enforced to be UTF8 in Rust

[dyedgreen]

Yeah, I think that’s where the problem lies. From what I can tell, files should be moved around as Vec<u8> rather than strings to avoid messing with their encoding. (At least for binaries like WASM).

I’m trying to get something like that to work, but I’m completely new to rust though, so the file fetcher and http_util thing is quite confusing 😅