Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Listing and extracting entries of a very large archive #564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
struffel opened this issue Apr 16, 2025 · 2 comments
Closed

Listing and extracting entries of a very large archive #564

struffel opened this issue Apr 16, 2025 · 2 comments

Comments

@struffel
Copy link

I have the following code for listing the contents of an archive given its url:

const zipReader = new zip.ZipReader(new zip.HttpReader(fileUrl));
const entries = await zipReader.getEntries();
console.log(entries);

This works well for small and medium-sized files, but the getEntries() it fails and creates a very ambiguous failed to fetch error for large archives.
The threshold seems to be at about 2GiB. This seems to match the threshold for blob sizes in Google Chrome.

Could this be the reason for the failure and is what I have even the right approach when trying to read large files on the client-side?
I have tried a few of the HttpOptions but enabling chunking had the side-effect of creating a ridiculous number of requests with 500kb each which is way to small and has a severe performance penalty when trying to actually extract the files (which is the intended next step).

Sample link for testing:
https://f003.backblazeb2.com/file/ambientCG-Web/download/Ground054_5jRbQ1OF/Ground054_12K-PNG.zip

@gildas-lormeau
Copy link
Owner

gildas-lormeau commented Apr 17, 2025

This is due to the fact that the entries are located at the end of the file. If the server supports range requests (see curl command below) then you just need to replace HttpReader with HttpRangeReader in your code to fix the issue. You can increase the chunk size (i.e. "500kb") by calling zip.configure({ chunkSize: <desired size in bytes> }).

curl -I https://f003.backblazeb2.com/file/ambientCG-Web/download/Ground054_5jRbQ1OF/Ground054_12K-PNG.zip
HTTP/1.1 200 
[...]
Accept-Ranges: bytes
[...]

@struffel
Copy link
Author

Thanks for this tip about configuring chunk size, I will try that out.

Repository owner locked and limited conversation to collaborators Apr 22, 2025
@gildas-lormeau gildas-lormeau converted this issue into discussion #565 Apr 22, 2025

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants