No way to continue without a test suite

[NikolajSchlej]

Long ago, when trees were big and bits were played live, I've developed those parsers in Qt-flavored C++, using lots and lots of different EFI and UEFI images as possible input, and adding a lot of really strange code to make most of them parsing into something resembling a valid tree structure.

Sadly, today in 2022 almost none of those original images are available to re-test that old code, so refactoring it got unnecessary complex, enough to discourage any new people to ever touch anything here. I myself was discouraged enough for many years.

To make this pain a lot easier, we seriously need to:

• write a simple test program that can be executed before and after any code change to confirm that the change did not break anything that already has a test for
• write a ton of tests for this program
• (re)collect some wacky EFI/UEFI images to test the corner cases and document those corner cases in the code (i.e. // This is to parse test image X, Y and Z)
• once all of the above is done, add testing in CI/CD pipeline so robots can do it for us
• then require tests to be written for any new PRs to be merged

[NikolajSchlej]

This image has a treasure trove of Phoenix NVRAM things, can be used to test that part of NVRAM parser later.
DA0V07MBAD1 REV D.BIN.zip

[platomav]

(re)collect some wacky EFI/UEFI images to test the corner cases and document those corner cases in the code (i.e. // This is to parse test image X, Y and Z)

Would it be possible to have a list of these edge cases that are historically supported so that anyone who has or knows how to re-find such samples can do so?

[NikolajSchlej]

Let's make such a list by reading parser sources:

• Capsules:

  • UEFI Capsule
  • AMI Aptio Capsule
  • Toshiba Capsule (Capsule GUID 3BE07062-1D51-45D2-4583-2BF093257ED461)

• Intel descriptor-based images:

  • Descriptor v2 (default, Skylake+)
  • Descriptor v1 (detected using componentSection->FlashParameters.ReadClockFrequency == FLASH_FREQUENCY_20MHZ check)
  • Descriptor with Gigabyte quirk (biosRegion.size == intelImage.size)

[NikolajSchlej]

Intel regions:

• Intel image with PDR region having FFS volumes/files (Apple x86 boards of 2015-2016)
• Intel image with DevExp1 region having ME FPT stuff (Recent Atom-based boards)
• Intel image with the last volume in Bios region intersecting with DevExp1 region (Also some recent Atom-based boards)

[NikolajSchlej]

Raw Areas:

• Intel image with a microcode in Raw Area
• Intel image with BPDT structures (IFWI 1.6, IFWI 1.7, IFWI 2.x)

[NikolajSchlej]

Volumes:

• Image with FSSv2 Volume Rev1 (Intel stuff before UEFI, older than P67 chipset, early Phoenix faux-EFI images)
• Image with FFSv2 Volume Rev2 (default)
• Image with FFSv3 Volume Rev2 (some newer stuff requiring more than 16 Mb of files in the volume)
• Image with FFSv2/v3 Volume Rev2 with extended header
• Image with Microcode Volume (Apple x86 before T2 2011-2015)
• Image with VSS NVRAM Volume (Phoenix, Apple)
• Image with AppleCrc32 and UsedSpace in Volume ZeroVector (Apple x86 before T2 2011-2016)
• Image with Weak-aligned Volume (haven't seen one yet, AFAIR)

[NikolajSchlej]

FFS Files:

• Image with files having non-empty tail (FFSv2 Rev1 volumes only, Intel stuff before UEFI)
• Image with large files (FFSv3 Rev2)
• Image with files having NVAR variable stores (AMI Aptio)
• Image with files having Phoenix hashes (Phoenix with BootGuard enabled)
• Image with files having AMI hashes (old and new, AMI Aptio with BootGuard enabled)

[NikolajSchlej]

FFS Sections:

• Image with padded compressed sections (Detect out-of-spec size requirement for some FFS sections #178)
• Image with Phoenix or Insyde postcode sections
• Image with EFI1.1/Tiano/LZMA/LZMAF86/GZip-compressed sections (EFI1.1 - early Intel before UEFI, GZip - Qualcomm arm64)
• Image with signed sections (EFI_CERT_TYPE_RSA2048_SHA256_GUID and EFI_FIRMWARE_CONTENTS_SIGNED_GUID)

[NikolajSchlej]

Still a ton of stuff remain for variations of NVRAM, ME, BPDT, BootGuard and other smaller parsers, but let's collect the above first and see how it goes.

[platomav]

AMI Aptio Capsule_rar.zip
Descriptor with Gigabyte quirk_rar.zip
Image with FSSv2 Volume Rev1_rar.zip
Intel image with BPDT structures - IFWI 1.6_rar.zip
Intel image with BPDT structures - IFWI 1.7_rar.zip
Intel image with BPDT structures - IFWI 2.0_rar.zip
Intel image with DevExp1 region having ME FPT stuff_rar.zip
Intel image with PDR region having FFS volumes_rar.zip
Intel image with the last volume in Bios region intersecting with DevExp1 region_rar.zip
LZMAF86_rar.zip

[NikolajSchlej]

Yet another breakage that could have been caught by a test: 19ab607

[NikolajSchlej]

Mentioning #219 here, have some images attached there.

[platomav]

Mentioning #234, which has a false positive "Intel" Flash Descriptor detection sample.

[NikolajSchlej]

Mentioning #368, regression in A66 and A67 that could be caught by adding Apple SCAP files to a test suite.

[platomav]

Sample for custom Microsoft LZMA GUID-defined section

• Unknown section type 00 instead of LZMA #416

[platomav]

Here is a sample for the AMD ZLIB GUID-defined Section (EFI_GUIDED_SECTION_ZLIB_AMD, CE3233F5-2CD6-4D87-9152-4A238BB6D1C4):

• https://www.asrock.com/mb/AMD/A520M-HVS/#BIOS

For the GZIP one (EFI_GUIDED_SECTION_GZIP, 1D301FE9-BE79-4353-91C2-D23BC959AE0C), I managed to find a sample within a Pixel 3 bootloader image:

• bootloader-blueline-b1c1-0.1-4948814.zip
• bootloader-blueline-b1c1-0.1-4948814.img_uefi.zip

Question for @NikolajSchlej :

UEFITool has references to another AMD ZLIB GUID (EFI_GUIDED_SECTION_ZLIB_AMD2, 991EFAC0-E260-416B-A4B8-3B153072B804), as well as an "AMD_COMPRESSED_RAW_FILE_GUID" (20BC8AC9-94D1-4208-AB28-5D673FD73487). Both of these were added at a7cf9cc (June 25, 2023) but, as far as I can tell, these are not used/implemented. Also, I cannot find any sample(s) for them (looked in Issues/PRs and googling). Do you happen to know where I can find samples for these (orphan) GUID definitions?

[NikolajSchlej]

@platomav, both are from AMD-based files for recent Ryzen machines, like this one.
JFP30.bin.zip
However, the current implementation is indeed incomplete because it requires adding parsing of RAW file (which UEFI PI spec specifically asks not to parse, for there be dragons). I've started to port this code from Binarly's fork, but then got distracted and it went nowhere since. yeggor@a6ef3a0

We need a discussion on parsing RAW files possible consequences of that, because @Ykidia also suggested adding several RAW file parsers in his recent PR.

[platomav]

Thank you Nikolaj!

Indeed, it will become necessary to start parsing non-standard areas (non-UEFI, Raw, Padding etc) for stuff which should not be there but unfortunately is. The safest addition is FV detection (due to the signature and various validations). But it may also require some magic to detect Files and/or Sections. There are known cases where vendors place Computrace modules within non-UEFI data. I've even seen FVs within PE32 Sections recently, although this last one is probably not something that is expected to be parsed (part of the binary's data, perhaps). This topic requires its own ticket/issue for sure.

Attaching here samples for guid-defined sections related to compression, so that we have them in one place:

• ZLIB AMD 1
• ZLIB AMD 2
• GZIP
• LZMA MS
• LZMA HP