Support chunk lazy loading

[brenogazzola]

Our production app using Webpacker 6 currently has many files that employ lazy loading of chunks:

// Only load Trix when necessary
  async bootstrap () {
    const { default: Trix } = await import(/* webpackChunkName: "trix" */ '../../chunks/trix')
    return Trix
  }

We are switching to jsbundling, but the code above is broken. It seems that Webpack can't find the chunk because of the fingerprint that Sprockets add to the end of the file name.

Is this possible with jsbundling right now, or would it be necessary for me to open a PR to add the support? I tried reading through the Webpacker code, but couldn't figure out what it was doing to make this work.

[dhh]

There's not a straight path I can see for relying on Webpack's internal chunking with this approach. You'll have to split out the lazy-loaded elements into their own entry points. That's a bit of a trade-off with this model.

[brenogazzola]

There's an alternative: Disable Sprockets fingerprinting and rely on the bundlers' content hash. Webpack and Esbuild have them, but I'm not sure about Rollup.

I don't think we currently have way to tell Sprockets to skip the digest of specific file types, but I think we could add a skip_digest_for configuration option that takes an array of file extensions. Then we can change the stats_digest method in the PathDigestUtils file to check that configuration:

def stat_digest(path, stat)
  if stat.directory?
    # If its a directive, digest the list of filenames
    digest_class.digest(self.entries(path).join(','.freeze))
  elsif stat.file?
    # If its a file, digest the contents
    digest_class.file(path.to_s).digest unless Sprockets.skip_digest_for.include?(File.extname(path))
  else
    raise TypeError, "stat was not a directory or file: #{stat.ftype}"
  end
end

With this the files will still have a different name if they changed, keeping the expected functionality, Sprockets will still add them to its manifest file, but any bundler feature that relies on the file name (like lazy loading chunks) will not break.

[dhh]

Please do investigate that! I like the idea that Sprockets could even auto-detect a pattern, and just skip files that have already been digested.

[brenogazzola]

I considered auto detection, but thought it might be considered too brittle.
I will take a stab at this and submit a PR to sprockets when I get something working.

[dhh]

I'd say we'd want to just auto-detect that the digest format matches what sprockets is already generating. And then we'll recommend/configure webpack to produce in the same format (same length/charset).

[brenogazzola]

Here it goes: rails/sprockets#714.
Took me half a dozen tries to figure out where to plug the behavior without breaking something else 😅

[dhh]

What's needed for the default webpack config to produce chunks with hashes in them? Also, I vaguely remember that if you digest hashes, there's an issue with the webpack dev server?

[brenogazzola]

This should config should match what sprockets currently does:

output: {
    path: path.resolve(__dirname, 'app/assets/builds'),
    filename: '[name]-[contenthash].js',
    chunkFilename: '[name]-[contenthash].chunk.js',
    hashFunction: 'sha256',
    hashDigestLength: 64
},

The current version of sprockets, assets:precompile generates these double digested files:

public/assets/fudgeballs-3e05d90ab38c43e41a3f1400bbae046de14b44df67edf0506fa0d68d744eaf20-9c3018226817bce637b2a39a98e52a3c818cd7e0ed397ed530353373919744e5.js
public/assets/fudgeballs_active_storage-8ff22166fa4c88fb20d8dfe95327a62decbd85690571a8ed9bffa261eb30ac8c.chunk-86b6f708a1df94cf6bf29f7177b184f6a86037d4bf396a0659d0dbc6b2a84931.js
public/assets/fudgeballs_clipboard-8ff22166fa4c88fb20d8dfe95327a62decbd85690571a8ed9bffa261eb30ac8c.chunk-701271f31e3fb24353d1e9e50a24d127f0391befcf5e13fe3074467e6304befa.js

The PR version correctly skips the fingerprinting:

public/assets/fudgeballs-3e05d90ab38c43e41a3f1400bbae046de14b44df67edf0506fa0d68d744eaf20.js
public/assets/fudgeballs_active_storage-ba5fc1e55a50a4647ee6e65f8270a22ddec5cf72618fbd5c71088cb0f2b9065d.chunk.js
public/assets/fudgeballs_clipboard-8ff22166fa4c88fb20d8dfe95327a62decbd85690571a8ed9bffa261eb30ac8c.chunk.js

There is however one configuration that Webpacker 5/6 has that I'm still missing and will try to find. This is the runtime.js that it produces, in order to figure out the correct path to the chunks:

n.u = e=>"js/" + {
  712: "fudgeballs_active_storage",
  875: "fudgeballs_clipboard",
  998: "trix"
}[e] + "-" + {
  712: "de39e19b81af6079c9306c79ae331ebc",
  717: "9957ff62fb2b6e6cbc110ad795b009f4",
  998: "4afa271d098ab9c491abfbf4838c7bf2"
}[e] + ".chunk.js",

This is what cssbundling with the config above is producing:

i.u = t=>({
  712: "fudgeballs_active_storage",
  717: "fudgeballs_invitations",
  875: "fudgeballs_clipboard"
}[t] + ".js")

So I've figured out how to produce sprockets like digests, but I still need to figure out how to tell Webpack to properly compile the js code with the digest.

[brenogazzola]

I'm not sure what you mean by 'digest hashes', but Webpacker 5/6 are working out of the box, no extra configs needed and the dev-server fine. We've been in this setup for months and both our local and production environments have to trouble finding the chunks.

The cssbundling gem has no problems in local environment using yarn build --watch. And production is what we are trying to fix.

[dhh]

Ah. Sweet. So you’re all good with your fork if sprockets with the patch?

[brenogazzola]

Yup, it's all working now: #21 👍