256 MiB maximum offset limit is too low #396

randomascii · 2024-03-18T16:40:41Z

The official dcheck enabled builds for Chrome are huge and have maximum section offsets of around 320 MB which is beyond the arbitrary hard-coded limit of 256 MiB in pefile.py. This causes symbol uploads to fail for these builds.

The limit should be increased to avoid this, and arguably the handling of the maximum should be changed so that an exception is thrown because the warning messages that are printed when invoking pefile.py are suppressed or otherwise hidden when using pefile.py as an import.

I have created a local change for Chrome that centralizes and increases the limit to 1 GiB.

In an attempt to detect corrupt files pefile.py imposes an arbitrary maximum offset of 0x10000000 (256 MiB). However Chrome's dcheck official builds exceed that limit which means that uploading of symbols started failing. This change increases the limit to 1 GiB. An issue has been filed against the pefile project for a long-term fix but this fix will unblock dcheck official builds for now. erocarrera/pefile#396 Bug: 329661971 Change-Id: Ica4905a61216a11e4ad56734977582133bc26684 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5377931 Commit-Queue: Bruce Dawson <[email protected]> Reviewed-by: Alex Gough <[email protected]> Cr-Commit-Position: refs/heads/main@{#1274350}

randomascii · 2024-03-18T20:38:46Z

The change I made in Chromium's copy of pefile.py can be found here:

https://chromium-review.googlesource.com/c/chromium/src/+/5377931

It's pretty simple - I just centralized the maximum offset limit and then raised it. I could easily make a PR for this change for here, if that would be accepted.

Or, maybe the limit should just be removed - maybe it has served its purpose and no longer makes sense.

Or, maybe the limit should be a constructor parameter. I haven't checked to see how much work this would be but at a high level this would be a cleaner solution.

Finally, if the limit is reached then I think an exception should be thrown so that pefile.py doesn't fail in silent and mysterious ways.

Any thoughts on this before I create a PR?

pefile has an hardcoded limit to 256MB per section: erocarrera/pefile#396 When building an initrd with large firmware files and lots of kernel modules, this limit can be reached. Skip over those warnings.

When parsing large images warnings are raised if they go beyond an hardcoded and fixed value. This is often not enough for many use cases, such as parsing a large Unified Kernel Image. Add a parameter, defaulting to the previous value. Fixes erocarrera#396

bluca · 2025-01-30T16:38:25Z

I have opened a PR that adds a new parameter to let users configure the offset limit: #460

pefile has an hardcoded limit to 256MB per section: erocarrera/pefile#396 When building an initrd with large firmware files and lots of kernel modules, this limit can be reached. Skip over those warnings.

pefile has an hardcoded limit to 256MB per section: erocarrera/pefile#396 When building an initrd with large firmware files and lots of kernel modules, this limit can be reached. Skip over those warnings. (cherry picked from commit 32caed5)

pefile has an hardcoded limit to 256MB per section: erocarrera/pefile#396 When building an initrd with large firmware files and lots of kernel modules, this limit can be reached. Skip over those warnings. (cherry picked from commit 32caed5) (cherry picked from commit 87224a2)

pefile has an hardcoded limit to 256MB per section: erocarrera/pefile#396 When building an initrd with large firmware files and lots of kernel modules, this limit can be reached. Skip over those warnings. (cherry picked from commit 32caed550f5a81eb87d2e39bc83917df2898d844) (cherry picked from commit 87224a2) (cherry picked from commit 9141043)

pefile has an hardcoded limit to 256MB per section: erocarrera/pefile#396 When building an initrd with large firmware files and lots of kernel modules, this limit can be reached. Skip over those warnings. (cherry picked from commit 32caed550f5a81eb87d2e39bc83917df2898d844) (cherry picked from commit 87224a2) (cherry picked from commit 9141043) (cherry picked from commit 870c665)

bluca linked a pull request Jan 30, 2025 that will close this issue

Add max_offset parameter to configure hardcoded offset limit #460

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

256 MiB maximum offset limit is too low #396

256 MiB maximum offset limit is too low #396

randomascii commented Mar 18, 2024

randomascii commented Mar 18, 2024

bluca commented Jan 30, 2025

256 MiB maximum offset limit is too low #396

256 MiB maximum offset limit is too low #396

Comments

randomascii commented Mar 18, 2024

randomascii commented Mar 18, 2024

bluca commented Jan 30, 2025