[Feature] Discussion around default package manager choice

[csvan]

Yarn has had its first major release [1], and I think it should be taken under serious consideration as a de-facto replacement for NPM as the default package manager.

Yarn is compatible with NPM while being much more performant. It also supports a wide array of features and is under active and rapid development. It has great traction in the community, with almost 200k repos on GitHub using it to some extent. Defaulting to it will likely be beneficial both to the community and the Node ecosystem at large.

[1] https://code.facebook.com/posts/274518539716230

[vkurchatkin]

Duplicate of nodejs/node#9161

[oliviertassinari]

Maybe we could unlock #9161? It's frozen I can't upvote the issue. This way, you could gather more feedback from the community.

[refack]

/CC @nodejs/user-feedback

[oliviertassinari]

Well, looking at https://stats.yarnpkg.com/, yarn accounts for 0.02% of the downloads, it's probably too early.

[arcanis]

It became impossible for us to get stats since npm moved to Cloudflare, circa ~june 2018. Back then, Yarn had 48% of the package downloads, in constant augmentation from ~20% in April 2017. Your guess is as good as mine as to where we are now 🙂

[muuvmuuv]

I would like to get this re-opened because of the reason that alternatives become more popular and are faster than npm (besides all that was mentioned in the #9161 discussion). The title should be Allow users to choose a default package manager.

• npm (default)
• pnpm (my choice)
• yarn 2 (berry)

[jasnell]

I would definitely champion work in this direction. There are a couple important bits:

1. For the Node.js installers, users should be presented a choice of package manager to install. Which package managers to include in last list can be a sensitive topic. We would need some baseline criteria for adding a package manager to the options list.

2. For the tar ball distributions, we would need a script that effectively does the same -- providing the ability to select a package manager to download and install.

3. Rather than bundling the npm client into the nodejs/node source tree, it should be removed. It might still be that we provide a distribution of the client, but it would not exist in the source tree.

[gireeshpunathil]

I am in favor of a user-selectable package manager, with the default one that is:

• better debug-able (error logs alone sufficient to figure out what went wrong)
• better documented (features easily understood and align with what it does)
• better modularized (less side effects and dependencies)
• well defined caching, and environment variables

than others. In addition, visible evidence of / promise on:

• an LTS policy aligning with node.js
• an open governance and community based project development and support model
• well defined and documented release cadence
• an agreed upon secure engineering policy

[gireeshpunathil]

re-opening, as the linked issue (to which this is a duplicate of) is closed and locked.

[arcanis]

One important question that we should first agree on is: what workflow do we want to improve? There are two different ones, and the solutions aren't the same for both.

1. Either we want to make it easier for each developer to create and work on their projects (so the solution would be to make it easier to work with a package manager of their choice),

2. Or we want to make it easier for all developers to work on all Node projects (so the solution would be to make it easier to work with any project, regardless of their package manager)

The difference may look subtle but let me show a practical example. VSCode uses Yarn as package manager (they have a yarn.lock checked-in, and they may use Yarn-specific features). Lodash uses npm (they have a package-lock.json, and they may also use npm-specific features). As such, it becomes clear that the choice of package manager isn't up to the users, but rather up to the projects! Indeed, the package manager (and its version) can be seen as the very first dependency for any project.

For this reason, I'm not convinced by, let's say, an install prompt that would allow choosing the package manager to install. In my mind, it'll simply fall into the same trap of making it harder to work with the package managers that didn't get picked. It may help new users a bit, but power users will end up still needing to install the other package managers manually.

So what do I suggest? I obviously really want to solve this. First, to summarize my ideal solution:

1. Package managers shouldn't necessarily be shipped by default,
2. I'd like projects to decide the package manager (not users)
3. I'd like the package managers install process to be transparent

Before going further, let me tell you about a feature of Yarn, something called yarnPath. When the global yarn CLI spawns, it looks for a .yarnrc.yml file into the current directory and greps it for a yarnPath key. If it finds it, it shells out to it. Basically, the idea is you can store your Yarn version in your repository, preventing cases where multiple members of a team use different versions of Yarn. Updating the version can be done through a dedicated command, yarn set version 2, which will download the latest release from our website, store it into the repository, and update yarnPath.

So .. what's my point? Simple: this workflow proves that we don't need the full package managers to be shipped within Node. What we do need are the binary names to be shipped with Node. For example for Yarn, all we really need is for a global yarn binary to support this yarnPath and yarn set version logic. If we have this, then it becomes up to each project to select the version they want to use, and we'll tick everything on the checklist:

1. The full Yarn binary wouldn't be shipped - for Yarn we would provide a few-KB source file that would only support the yarnPath and yarn set version code path, and for npm/pnpm I believe we could write a very minimal wrapper that would follow the workflow I showcase below.

2. Each project would be able to define the version of the package manager they want to use, and having multiple projects with multiple different versions wouldn't be a problem.

3. Most users wouldn't even have to be aware of this, because the details would be abstracted out.

And it's not Yarn-specific! Generally speaking, all we need is to mock the package managers binaries to "jumpers" - very simple binaries that would throw an error if the version they detect should be used isn't installed. For example, here are a few examples of the workflow I have in mind:

$ mkdir my-project && cd my-project

$ npm install
No npm selected. Do you want to use 6.14.1 (latest)? [Y/n]
Installing npm 6.14.1... Done.
Installing the project... Done.

$ npm --version
6.14.1

$ jq .engines package.json
{
    "engines": {
        "pm": "[email protected]"
    }
}

$ git clone [email protected]:arcanis/my-pnpm-project my-pnpm-project
$ cd my-pnpm-project

$ jq .engines package.json
{
    "engines": {
        "pm": "[email protected]"
    }
}

$ pnpm install
Installing pnpm 4.11.1... Done.
Installing the project... Done.

$ pnpm --version
4.11.1

$ pnpm install
Installing the project... Done.

$ git clone [email protected]:arcanis/my-yarn-project my-yarn-project
$ cd my-yarn-project

$ cat .yarnrc.yml
yarnPath: .yarn/releases/yarn-2.0.0.js

$ yarn --version
2.0.0

$ yarn install
Installing the project... Done.

[codepunkt]

I like your proposal. However, the user still has to know the appropriate CLI command for the package.json's engine/pm field.

What if we had a generic command for that aswell, e.g.:

node packages install
# checks package.json, installs pm engine if not available, uses it to install deps

[arcanis]

What if we had a generic command for that aswell, e.g.:

There are a few ways we could solve it. For example, we could simply make the jumpers warn when used on incompatible projects:

$ jq .engines package.json
{
  "engines": {
    "pm": "[email protected]"
  }
}

$ npm install
Error: This project is configured to be used with pnpm.
Please run `pnpm install` instead.

This way each project would keep their identity, and users wouldn't be confused as much by a command having different behaviors on different projects (my main worry with having a single command).

[muuvmuuv]

LGTM @arcanis ! What about global installed packages? Some prefer npm, some prefer yarn and if both are used, incompatibilities could appear (had this issue with NVM, NPM and pnpm). To your suggestion I would add this:

Users can select a default package manager, this would be picked by default for global packages and newly created projects. For example, when I create a new project I now do node init which creates a package.json and installs the global default version into my project. Then we are no longer do npm install oder yarn add but more likely node add eslint and node add -g eslint which would automatically select the correct package manager to use.
Node would need to add command aliases like @ionic/cli does it.

Additionally, I would cache and symlink the PM versions, so the user is not downloading duplicated versions, when they already got the same in another project.

[arcanis]

While aliases might be useful in the future, I think this should be a follow-up and considered out-of-scope for this discussion. Multiple reasons for that:

• Having node aliases isn't a prerequisite for having the regular binary jumpers (whereas having the jumpers is a prerequisite for having aliases).

• I suspect adding commands to the node binary will be a tougher sell, so I'd prefer to avoid this part of the discussion (the worst case would be that it reaches a dead end, blocking as a result the whole idea or at least delaying it significantly).

  • Another angle is that I suspect there is no right or wrong position on this, so keeping the discussion open and productive might be difficult. By comparison I think everyone is now aware that the npm binary isn't the exclusive JS package manager anymore, so we'll be able to spend more time on the "how" than debating the "why".

• It's not clear how much would be aliased, so more discussion would be needed. Only install? Or add too? What about --version? Or workspaces info? What about the CLI options? (No need to answer, it's just an example of the kind of complexity that this would add)

• There are identity concerns with aliasing the package manager versions under common names. For example, let's say that node install crashes, where do we expect users to fill the bug reports? Etc, etc.

I'd prefer we stay focused on global binaries, and later on we'll have the time to think about a Node integration - in a separate project dedicated to this goal.

[bnb]

I have two follow-ups on @arcanis’s proposal:

1. What makes a package manager “good enough” to ship a stub with Node.js? For example, if user X comes in and asks us to ship a Bower stub, should we? What about the 100 other new package managers this will spawn? Where do we draw the line and how does that evolve over time?
2. What happens when a package manager is unmaintained, dead, or otherwise undeveloped? How do we manage the legacy stubs? Are we stuck with them forever, like we are with many OG bits of core?

Looking into the future I see exponential growth of projects that are added to this, leading to a more fractured ecosystem and overall worse DX for new and novice users. I am extremely cautious about shoving this burden on users.

Additionally, to both @jasnell and @arcanis, how do you envision your potential solutions being effective for FaaS platforms, Docker images, and PaaS platforms without tremendous additional overhead? (I am aware that this has been a request from FaaS folks who are running code that doesn’t need npm!)

[zkochan]

Another idea that I think wasn't mentioned. Maybe it needs to be a package manager manager + node manager. Maybe it should be done on the nvm/nvs level as different projects are also expecting a specific version of Node, not only a specific version of the package manager.

developing packages is a big enough use case to make node ship a package manager manager

I think it is less about developing packages and more about developing large enterprise projects. Small package authors don't really care about which package manager they use. Large projects might need specific features of specific package managers.

[jasnell]

@arcanis ... that's awesome. Give me about a day to have time to check out your POC then get back to you on it.

@devsnek ... I definitely agree with what you're saying but currently, it's just not clear who actually owns the definition of How a Node.js Module Actually Works. I think we need to put some thought in to addressing that.

[mcollina]

I would be highly in favor of having stubs that install them on demand.

[NonPolynomial]

I read the entire thread and first of all I'm thankful, that this discussion has started because I think it is really important to have this discussed.
An I'm thankful that so many people contribute to this discussion with opinions and possible solutions.

To give my two cents to this discussion:
I'm just a casual user of npm and would love to use yarn per default in projects.
BUT in my opinion should Node.JS shipped with npm per default (or as default package manager - if you like to describe it that way), simply for that reason, that NPM is not only a package manager but also the registry that is used by all other (bigger) package managers per default.

[jimmywarting]

How about no package manager at all like both Deno and browser handle it? With absolute http urls!
Would be nice to just import x from 'https://example.com/lib.js' without having to install anything and supporting lazy loading/caching as needed.

Obvious we would need more security options like disable/sandboxing fs, net, process & env
wish i would like to have either way (whatever we decide to support)

[octogonz]

Would be nice to just import x from 'https://example.com/lib.js' without having to install anything and supporting lazy loading/caching as needed.

What is Deno's story for versioning? How do you distinguish SemVer major/minor/patch bumps? How do you express a peer dependency relationship?

[octogonz]

Also, suppose:

• A depends on B and C
• B depends on D@^1.0.0, but
• C asks for D@~1.2.3

How would Deno ensure that a single copy of [email protected] gets shared between B and C, instead of side-by-side installations of [email protected] and [email protected]? This is what package managers are all about.

[jimmywarting]

That can be handled by some CDN servers url structure
Eg https://unpkg.com/[email protected]/dist/cjs.js

Googles Hosted Libraries have done it even since before node's was introduced

denoland/deno#47

Import maps is also possible in deno that can help with dependencies

[isaacs]

@jasnell

There is another aspect to all of this that is important and that has not yet been discussed. Yarn 2 introduces the pnp concept of package resolution. npm, Inc. has indicated that their future directions around tink also intend to move in that direction, and while there is a good proportion of the ecosystem that appears to favor that model, a number of major enterprise users of Node.js actually do not -- and do not want to see pnp become the default model for package management in Node.js. Should a decision be made here to stick with npm as the default client, we'd need to have a discussion around this regardless of any other clients being used by the ecosystem if the intent is still to move to the tink/pnp approach.

In other words, Node.js core needs to come to agreement around one very specific question: What do we (Node.js core) wish to define as being the minimal default package management experience out of the box with Node.js core. I would argue strongly that pnp is not it. Node.js core should provide a minimal experience for installing and publishing modules. Nothing more, nothing less.

There's a pretty fundamental difference between tink and pnp, afaik. Tink does not change or override the Node.js module resolution process. Instead, it creates a virtual filesystem, such that the Node.js module resolver just works like it always has. When it gets ENOENT errors from fs operations, these operations are re-tried against the appropriate file found in a centralized cache.

We fully appreciate that enterprise (or other) users may prefer to install without this kind of experimental magic. When we do implement it (which won't be for a while yet), it'll be opt-in for at least one major release, and assuming all goes well, opt-out thereafter. We have no plans to ever ship a version of npm that can't install your dependencies right into node_modules as normal files, and may back away from this entire endeavor if it turns out to be a bad idea in practice.

npm is at least as committed to backwards compatibility as Node.js, even to a fault. And we're of course happy to keep the lines of communication open and make sure we're on the same page about the DX that npm and Node.js provide for our users.

[pauldraper]

There's a pretty fundamental difference between tink and pnp, afaik.

The biggest difference is that tink is dead.

[meadowsys]

For the time being this is available, and you can add this to your package.json to stop the wrong manager from being used:

{
   "scripts": {
      "preinstall": "npx -y only-allow pnpm" // for example
   }
}

But this only has (according to the readme) npm, pnpm, and yarn options, and I see this as more of a hack. (and npx is used, lol)

I think I am for the stub / choose in installer option. If these don't work, honestly I think the current bundling of npm is fine, since you can install all other managers once you have one.

the community of pnpm users really dislike being ignored or left out of consideration on these things :-)

@jasnell haha yea

[ljharb]

you can do npx --yes only-allow npm or whichever.

[meadowsys]

another thing: a package manager is a package manager manager since a package manager is just a package. and I dont think we need a dedicated package manager manager

[csvan]

I created this issue over 3 years ago, when Yarn still had a strong advantage over NPM in several respects. Since then, NPM has come a very long way (especially with the release of v7) and I personally no longer support this motion.
In fact, at my current workplace we are in the process of migrating back to NPM from Yarn, since it lets us reduce the overall complexity of our stack by cutting out a third-party package manager, while now getting practically all the benefits we got from Yarn.

[joshmanders]

@csvan I'm in the same boat. npm v7 is amazing.

[pauldraper]

npm still doesn't support resolutions which are GOLDEN for addressing security vulns and other problems.

IDK how people live without it.

It's okay. npm will get around to it in a couple years. Along with PNP, npm why, etc.

[csvan]

Not OOTB unfortunately, but there is this: https://www.npmjs.com/package/npm-force-resolutions

[darcyclarke]

We're actually looking to implement overrides soon (ie. towards the end of this - or early next - quarter) which should help resolve this. As for npm why... we actually have npm explain now. Virtual filesystem/resolution strategy is on the radar (ref. prior art being tink) but obviously a lot of discussion/design work to be done there.

[isaacs]

Also, npm why is an alias for npm explain

[muuvmuuv]

I am using npm ls webpack at the moment, and it works fine.

[muuvmuuv]

[arcanis]

That has nothing to do with the problem at hand, or even package managers in general. The MS quote that mentions them is erroneous, and I believe its author meant to say "package registry" instead of "package manager" (or at least not the JS ones; I'm not an expert in other languages, and it's possible they have dangerous defaults that compound the problem).

In fact, as far as the JS ecosystem is concerned, package managers such as Yarn or npm all store both package versions and checksums within the lockfiles, protecting against these attacks. It seems the affected JS projects didn't use lockfiles (perhaps due to the annoying "don't use lockfiles on libraries" cargo cult?), but that goes against the workflow our tools explicitly advocate.

To reiterate, whether the remote registry url is contained within the source code or the project's .yarnrc.yml has literally zero incidence on this kind of problems - the same thing would have happened to anyone writing Deno code with one of the misbehaving registries.

[muuvmuuv]

I agree, yes it is about the registries but npm is the js public registry in that case. A Lockfile can protect you but not in newly created protects, so if the attacker is already aware of your code and name it would be super easy to fake its functionality.

In case of Deno you target a specific package directly, so it would only be possible if you change its source code. If an employee puts in some public registry URL he should be smart enough to see it's not their companies URL he's typing.

[arcanis]

In case of Deno you target a specific package directly, so it would only be possible if you change its source code. If an employee puts in some public registry URL he should be smart enough to see it's not their companies URL he's typing.

That's incorrect. The problem was that the proxy registries (JFrog, Azure, etc) were returning the incorrect packages by serving the public ones over the private ones. In other words, a trusted server (the proxy) was taken advantage of to return malicious data. The url doesn't have to be modified in the source, since it's still the same one before and after compromission.

Since enterprise users, Deno or not, will always use a proxy over raw http dependencies, the problem would be the same: if the proxy returns the wrong data for your requests, then at least you better have a package manager between you and it to detect such problems as soon as you can.

[jimmywarting]

I also touched this subject earlier a few post above, I like the idea of no package mananger. it's faster if you want to start prototyping and don't require you to install anything + stuff can be lazy loaded as needed with async import. So you wouldn't have to download everything. But however to tackle this security issue you need some integrity check there is a few topics on how this is getting tackled in both Deno and Browser spec wise.

• Subresource integrity for modules denoland/deno#200

[muuvmuuv]

Uff. You are right I misunderstood this, sorry.