Node compatibility

[ry]

We've resisted considering Node compatibility as Deno was under initial development so that those considerations didn't influence the design. But now Deno has a relatively established API and the URL module system is deeply baked in. It is unlikely, at this point, that considering compatibility with existing Node modules will compromise Deno's design goals.

It's unlikely we can make Node code work 100% of the time. But we can do a lot to make porting modules over easier. Let's discuss how make Deno work with existing Node code. Here are some ideas:

• One direction could be a transpilation program that takes a CommonJS node file, and translates it into ES modules.

• Node has core APIs like file system and networking which are different (sometimes subtly so) in Deno. For example, require('fs').readFileSync() in Node supports encoding to strings, Deno.readFileSync() does not. I suggest we implement a compatibility layer in deno_std/node to match the semantics. (There has been some effort to do this already in https://github.com/denolib/node, but progress is slow, I think it makes sense to do it in deno_std.)

• We should attempt a require() implementation. It should probably be in deno_std rather than deno itself. It might make sense to base this on top of dynamic import Support dynamic import #1789 (which will land very soon - see Dynamic import #2516).

Existing work

• https://github.com/denolib/node-to-deno
• https://github.com/denolib/node

[zekth]

I've started to work on https://github.com/denolib/node with @kevinkassimo but this feature was not having so much following untill now so not so much effort has been involved it. If you want to make it part of deno-std i can address the port of core APIs.

[kitsonk]

@ry my suggestion is to scope what the expectations around require() are. It easily becomes a complicated thing, because of the rather complex module resolution logic (as I know you are aware), but it is resolution logic that lots of Node.js code relies upon. There are also the decorations on require() which a decent amount of code uses to hack around with.

Another couple thoughts about compatibility is:

1. do you need to build it into Deno? Is there some preprocessing, potentially on the fly, that could be offloaded to an intelligent CDN? https://www.pika.dev/cdn takes ESM modules out of npm and bundles them for being loaded in the browser, resolving dependencies, etc. Something like that for Deno might be an easier path than building all the heavy lifting in the runtime engine.
2. Is there some way the TypeScript compiler do the heavy lifting. While it wouldn't re-write module specifiers, it can convert JavaScript CommonJS modules to ES Modules. If we implemented whatever module resolution logic we wanted to support in Rust land and used the compiler to simply rewrite the modules to ES, we wouldn't have to implement require. I just can't remember what happens when you have a require() not at the top level. 🤔

[fusionstrings]

This is great to have. I have been following JSPM since beginning of the project. In its latest iteration JSPM2 @guybedford has managed a great feat in this area and result is in terms of https://github.com/jspm/jspm-resolve. The overall scope of JSPM is big but a solution to module resolution is at the core.
I guess this is where deno might join hands with JSPM. A problem which has already been identified, researched and solved at great extend from all possible perspectives and keeping in mind limitations of the environment and infrastructure (network and web) by Guy. What’s left is to align it with this context.

I have (and I see several others) been using modules from https://jspm.io/ (which is same as using jspm resolver) in deno and it works out great.

If deno includes jspm resolver in std, it will be huge for deno and JSPM both because I assume both platforms want to align with web and tc39 standards as possible.

[guybedford]

Thanks @fusionstrings for pinging me on this issue. An https://deno.jspm.io service could certainly be an interesting prospect. Since jspm.io already does the heavy-lifting of converting CommonJS to ES modules conversion while supporting Node.js semantics and strict mode conversions, the only part of the problem left to solve to get this support would exactly be that compatibility wrapper for Node.js core libraries as is being worked on in https://github.com/denolib/node.

I think it would actually be quite easy to throw such a server together given enough coverage of the core APIs through the compatibility layer. I will continue to track the repo there, and happy to discuss further too.

[guybedford]

Also, if anyone is interested in working on this problem separately to jspm, the CJS -> ESM conversion layer that is used is open sourced here - https://github.com/jspm/babel-plugin-transform-cjs-dew.

[thisconnect]

One direction could be a transpilation program that takes a CommonJS node file, and translates it into ES modules.

library authors are already moving to shipping more and more native es modules

Node has core APIs like file system and networking which are different (sometimes subtly so) in Deno. For example, require('fs').readFileSync()

Node.js itself is moving to native esm i.e. import fs, { readFileSync } from 'fs';

https://nodejs.org/api/esm.html

The future will be without require nor any other custom non-standard

[szhu]

I'd love to see a npm library that implements the Deno api using the Node stdlib.

This library would:

• Allow people familiar with Node to start to enjoy some parts of Deno. This is good for people who can't immediately switch to Deno right away for some reason. This will also make it easier to convince users that Deno will be easy to work with if they do switch.

• Make it possible for simpler projects to support both Node and Deno without needing to transpile code. Also it makes the experience more pleasant -- if I need to write such code, I'd prefer writing Deno code and supporting Node as an afterthought using this library rather than writing Node code and supporting Deno as an afterthought using deno_std/node.

[kevinkassimo]

I'd love to see a npm library that implements the Deno api using the Node stdlib.

Here is one: https://github.com/keroxp/denode