Reproducible npm tarballs

[Krinkle]

We currently have a reproducible qunit.js release artifact.

It is byte-for-byte identical between what one can independently generate using npm run build (release doc), and what we have published to the npm registry and on code.jquery.com CDN.

As a next step, I'd like for downstream tarball created by npm to also be reproducible, and for the verification process to be automated from a nighly Travis CI cronjob (e.g. build latest release tag, and compare against downloaded tarball).

Some pointers to get going:

• npm tarballs are available from the URL pattern https://registry.npmjs.org/NAME/-/NAME-VERSION.tgz such as https://registry.npmjs.org/oojs/-/oojs-5.0.0.tgz for oojs@5.0.0
• Probably the only thing that might be non-deterministic is file timestamps and compression file order. However, I think both of these were already made deterministic by npm for other projects. I'm seeing all files consistenly have a timestamp in the year 1985, and the compression order appears to be consistently alphabetical. We just have to verify this, but otherwise probably nothing to do.
• See Travis CI - Cron jobs docs for setting up a nighly job.
• You can use git tag --sort=v:refname --list '[0-9]*' to find the latest tag, as sorted by SemVer.
• diffoscope might be of use.

See also:

• Prior art at Make QUnit releases deterministic (reproducible builds) #1538.
• Related thread at improve colors of project badge jvm-repo-rebuild/reproducible-central#54.

[Krinkle]

It turns out my hunch was correct. The npm cli has indeed already done all the heavy lifting in terms of file timestamps, file order in the archive and other glue in the tgz file, all being deterministic and reproducible.

When I check out 2.16.0 in a clean git clone of our repo, and run the build, and run npm pack (Node 14, macOS, npm 7.10.0) then the tgz file that gets created is byte-identical and matches the checksum of the tgz uploaded to the registry last month.

However, when I do the same from Linux (Node 12, npm 7.5.2), the extracted files are identical but the tgz checksum does not match. With the help of diffoscope the difference was very quickly found:

--- npm-2.16.0.tgz
+++ qunit-2.16.0.tgz
├── filetype from file(1)
│ @@ -1 +1 @@
│ -gzip compressed data, max compression
│ +gzip compressed data

This turns out to be due to a difference in zlib compression defaults between platforms. This was already reported and fixed upstream (npm/pacote#71, npm/cli#2922) and released as part of npm 7.7.0, so if we upgrade the reproducible-builds CI job from Node 12 to Node 14, we can probably add a simple npm pack step at the end and then add an assertion for its checksum to match.

[Krinkle]

I'd like to keep our CI simple and use only pre-installed Node+npm versions. This means either Node 12 or 14 with npm 6.x, or Node 16 with latest npm 7.x.

We can't use Node 14 with npm 6.x because npm-pack wasn't yet deterministic there (npm/pacote#71), which was fixed in npm 7.7.0.

But, updating the job to Node 16 (and thus latest npm 7.20), fails the build due to an unrelated bug in npm with v1 lock file processing being broken for the file:npm-reporter local dependency we have. I've reported this upstream at npm/cli#3628.

[Krinkle]

Locally, I use Node 14 with npm 7.5.2, and installation still works there, so it's presumably not an (intentional) breaking change in 7.0, but rather a regression bug introduced later in the npm 7.x series between 7.5.2 and 7.20.3.