Description
Feature Proposal
I was wondering if it would be possible to add support for parallel asset compression using Node.js worker threads, similar to what terser-webpack-plugin does. The idea is that this might help with performance and memory issues for larger builds, since each worker would use its own V8 heap and CPU core.
I’m not totally sure if this is the best solution, but it seems like it could help with the heap fragmentation and "out of memory" errors we’ve been seeing when everything happens on the main thread.
Feature Use Case
We’ve been running into "JavaScript heap out of memory" errors in our CI/CD pipeline during the compression phase.
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Looking into it, it seems like the current approach (compressing all assets on the main thread) is causing high memory usage and fragmentation, especially with lots of large assets.
To keep our builds running, we had to keep increasing the --max-old-space-size flag—from 8GB to 9GB, and then to 10GB, all within a 2 month period. This quick escalation made it clear that just increasing memory wasn’t a sustainable solution for us, especially as our codebase keeps growing.
Would using worker threads for compression be a good way to solve this? If so, it would be really cool to have this as an option in the plugin! As a workaround, we wrote our own batching script to reduce memory pressure, but native support for worker threads might be a more robust fix.
Please paste the results of npx webpack-cli info here, and mention other relevant information.
None