[BUG] package-lock.json integrity value for git dependencies depends on the architecture (Apple Silicon M1 differs)

[feross]

Current Behavior:

The package-lock.json integrity value seems to depend on the OS/architecture. Take the following git dependency which specifies a commit hash:

npm pack "git+ssh://git@github.com/jhiesey/idb-kv-store.git#109ccad165fd6470e12fd66025da9e4743a46043"

The integrity value produced is different on these OSes/architectures:

• Ubuntu 20.04, and macOS 11.2.3 (Intel): sha512-DnBTbDDxd9/9mwPehyraeuRTbNEqbWLcAdE3GC1trdBWWwKnkWsaU/X6mVLIKKB/IYWmG+cnL3ihg/Ql/rW5kg==
• macOS 11.2.3 (Apple Silicon): sha512-T3ZWOM1TT+Ch/splApkEe1HwktWs+n/iHvDvtIGEI+4xuMGHite6mMujuNd8sen49ofLP/PxzprQMSPJK8APww==

Expected Behavior:

The integrity value should not be different on Apple Silicon (M1 chip) machines.

Steps To Reproduce:

Run npm pack "git+ssh://git@github.com/jhiesey/idb-kv-store.git#109ccad165fd6470e12fd66025da9e4743a46043" and inspect the integrity value from an M1 Mac. Node.js was installed from Homebrew using brew install node and the amd64 version was installed.

Also... @jhiesey and I dug into this a bit and found that the tarballs fetched from the GitHub CDN are exactly the same on M1 and other architectures, byte-for-byte. Same for the ungzipped tarballs – they are the same byte-for-byte. What differs, though, is the gzipped tarballs (.tar.gz) files. Those appear to have substantial differences when viewed in a hex editor.

Environment:

• OS: Various, see above
• Node: v15.11.0
• npm: 7.6.2

[feross]

I've included both the Intel and M1 gzipped tarballs here, if that's helpful: Archive.zip

[reggi]

I have both M1 and Intel mac, let me know if I can be of help.

Here's the intel response:

Thomass-MacBook-Pro-2:~ thomas$ npm pack "git+ssh://git@github.com/jhiesey/idb-kv-store.git#109ccad165fd6470e12fd66025da9e4743a46043"
npm notice 
npm notice 📦  idb-kv-store@4.5.0
npm notice === Tarball Contents === 
npm notice 1.1kB  LICENSE          
npm notice 16.5kB idbkvstore.min.js
npm notice 14.5kB index.js         
npm notice 2.5kB  karma.conf.js    
npm notice 17.7kB test.js          
npm notice 1.2kB  package.json     
npm notice 10.4kB README.md        
npm notice 1.5kB  .travis.yml      
npm notice === Tarball Details === 
npm notice name:          idb-kv-store                            
npm notice version:       4.5.0                                   
npm notice filename:      idb-kv-store-4.5.0.tgz                  
npm notice package size:  16.5 kB                                 
npm notice unpacked size: 65.4 kB                                 
npm notice shasum:        f760c3e1f1ca0959863df9e56a0609cebdc941a3
npm notice integrity:     sha512-xAVBTqp3mAtgt[...]B/PjnXc8eSTBg==
npm notice total files:   8                                       
npm notice 

Here's the M1

thomasreggi@Thomass-MacBook-Air ~ % npm pack "git+ssh://git@github.com/jhiesey/idb-kv-store.git#109ccad165fd6470e12fd66025da9e4743a46043"
npm notice 
npm notice 📦  idb-kv-store@4.5.0
npm notice === Tarball Contents === 
npm notice 1.1kB  LICENSE          
npm notice 10.4kB README.md        
npm notice 1.5kB  .travis.yml      
npm notice 16.5kB idbkvstore.min.js
npm notice 14.5kB index.js         
npm notice 2.5kB  karma.conf.js    
npm notice 1.2kB  package.json     
npm notice 17.7kB test.js          
npm notice === Tarball Details === 
npm notice name:          idb-kv-store                            
npm notice version:       4.5.0                                   
npm notice filename:      idb-kv-store-4.5.0.tgz                  
npm notice package size:  16.5 kB                                 
npm notice unpacked size: 65.4 kB                                 
npm notice shasum:        bfbf05231b88deb3ec0ca8e9bf6ec02bbb96ac7d
npm notice integrity:     sha512-T3ZWOM1TT+Ch/[...]prQMSPJK8APww==
npm notice total files:   8                                       
npm notice 

[Myriachan]

A friend linked me this, and I just wanted to point out that relying on compressed data to match on all machines given the same uncompressed input is rife with problems. Implementations of Zlib exist that use hardware acceleration instructions, and they do not take care to ensure that each accelerated version will come up with the same compression as every other implementation.

As an example, there is a Zlib implementation that uses x86 SSE4.2 instructions to accelerate pattern matching, and ARM doesn't have anything like that, so this is an x86-specific code path. It does not find the same exact dictionary matches, so won't come up with the same results as the pure C implementation.

[feross]

A friend linked me this, and I just wanted to point out that relying on compressed data to match on all machines given the same uncompressed input is rife with problems

I also felt like this design decision might have this issue, even with the portable: true option that is passed to minizlib. I wonder why they're not instead using the .tar file for integrity purposes?

[darcyclarke]

@feross thanks for filing this - we'll take a look right away (apologized it wasn't triaged quicker)

[jaydenseric]

This bug had absolutely stumped me since december 2020, regarding this dependency tree:

npm ls bls12377js
[redacted]@ [redacted]
└─┬ bitgo@11.11.0
  └─┬ @bitgo/account-lib@2.8.0
    └─┬ @celo/contractkit@0.4.11
      └─┬ @celo/utils@0.1.17
        └── bls12377js@0.1.0 (git+ssh://git@github.com/celo-org/bls12377js.git#cb38a4cfb643c778619d79b20ca3e5283a2122a6)

I created the package-lock.json file locally, and when attempting to push it to Heroku it would result in the following deployment build error (log excerpt):

-----> Installing dependencies
       Installing node modules
       npm ERR! code EINTEGRITY
       npm ERR! sha512-7EgC8H/EFwDko7LjlkOQslnckOdsknH8zVPQ2s/4Gya3jXGvxFLqNhTFOiHkaz1mSxbC7dhEK4LbHJEjjBsfIA== integrity checksum failed when using sha512: wanted sha512-7EgC8H/EFwDko7LjlkOQslnckOdsknH8zVPQ2s/4Gya3jXGvxFLqNhTFOiHkaz1mSxbC7dhEK4LbHJEjjBsfIA== but got sha512-oUocNv3wT2CYJys8ZjPK5TfynQ81T7IfAmDn4ZmZIA7Hv/RdiD9HeDvVgTUUfUx8udJmYYh57A78zqdhKe6x7Q==. (47748 bytes)

Heroku support and I gave up trying to figure out the problem and for that GraphQL API I ended up downgrading to npm v6.

Running with Node.js v15.12.0 and npm v7.6.3:

npm pack "https://github.com/celo-org/bls12377js#cb38a4cfb643c778619d79b20ca3e5283a2122a6"

In macOS v11.2.1 (Intel chip):

npm notice === Tarball Details === 
npm notice name:          bls12377js                              
npm notice version:       0.1.0                                   
npm notice filename:      bls12377js-0.1.0.tgz                    
npm notice package size:  47.7 kB                                 
npm notice unpacked size: 144.3 kB                                
npm notice shasum:        13e4f2f388d4f8f28e0fa45a8ceaef4ca7514789
npm notice integrity:     sha512-7EgC8H/EFwDko[...]4LbHJEjjBsfIA==
npm notice total files:   33 

In Ubuntu 20.04.2 LTS (Heroku, not sure what chip):

npm notice === Tarball Details === 
npm notice name:          bls12377js                              
npm notice version:       0.1.0                                   
npm notice filename:      bls12377js-0.1.0.tgz                    
npm notice package size:  47.7 kB                                 
npm notice unpacked size: 144.3 kB                                
npm notice shasum:        616f7b4f14365eff44c82539994634ba94d33911
npm notice integrity:     sha512-oUocNv3wT2CYJ[...]A78zqdhKe6x7Q==
npm notice total files:   33

At first I noticed the file permissions were created differently for the dependency installed locally (drwxr-xr-x/755) vs installed in Heroku (drwx------/700), but it turns out that doesn't have an affect on npm pack shasum. I verified the contents of literally every file and they are identical locally and in Heroku. The npm shasums are literally platform dependant / incorrect, as per this issue raised 😕

This is a critical, show-stopping bug; how long until it's fixed in a stable npm release? Is there a workaround in the meantime, that does not involve downgrading npm v7 to npm v6? One gross possibility is not committing a lockfile and let Heroku generate one each deployment; the checksums can be whatever that way, since they're not compared against anything. Obviously that would have undesirable stability and security implications.