DATA_DIRECTORY missing entries for malformed values of NumberOfRvaAndSizes

[huettenhain]

I noticed during malware analysis that pefile could better handle certain cases where the value of NumberOfRvaAndSizes has been reduced to hide the .NET directory information from certain parsers. Such a change can be detected by comparing the SizeOfOptionalHeader field with the size of the optional header if the given NumberOfRvaAndSizes is assumed.

To reproduce the problem, take any .NET binary and reduce the value of NumberOfRvaAndSizes with a hex editor to 0x0000000E, say. I took a copy of RegAsm.exe and patched it:

>>> data = open('RegAsm.Patched.exe', 'rb').read()
>>> pe = PE(data=data)
>>> pprint(pe.OPTIONAL_HEADER.DATA_DIRECTORY)
[<Structure: [IMAGE_DIRECTORY_ENTRY_EXPORT] 0xF8 0x0 VirtualAddress: 0x0 0xFC 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_IMPORT] 0x100 0x0 VirtualAddress: 0xB78C 0x104 0x4 Size: 0x4F>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_RESOURCE] 0x108 0x0 VirtualAddress: 0xC000 0x10C 0x4 Size: 0x608>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_EXCEPTION] 0x110 0x0 VirtualAddress: 0x0 0x114 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_SECURITY] 0x118 0x0 VirtualAddress: 0x0 0x11C 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_BASERELOC] 0x120 0x0 VirtualAddress: 0xE000 0x124 0x4 Size: 0xC>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_DEBUG] 0x128 0x0 VirtualAddress: 0xB720 0x12C 0x4 Size: 0x1C>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_COPYRIGHT] 0x130 0x0 VirtualAddress: 0x0 0x134 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_GLOBALPTR] 0x138 0x0 VirtualAddress: 0x0 0x13C 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_TLS] 0x140 0x0 VirtualAddress: 0x0 0x144 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_LOAD_CONFIG] 0x148 0x0 VirtualAddress: 0x0 0x14C 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_BOUND_IMPORT] 0x150 0x0 VirtualAddress: 0x0 0x154 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_IAT] 0x158 0x0 VirtualAddress: 0x2000 0x15C 0x4 Size: 0x8>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_DELAY_IMPORT] 0x160 0x0 VirtualAddress: 0x0 0x164 0x4 Size: 0x0>]
>>> nt = pe.DOS_HEADER.e_lfanew
>>> fix = data[:nt+0x74] + b'\x10\0\0\0' + data[nt+0x78:]
>>> pprint(PE(data=fix).OPTIONAL_HEADER.DATA_DIRECTORY)
[<Structure: [IMAGE_DIRECTORY_ENTRY_EXPORT] 0xF8 0x0 VirtualAddress: 0x0 0xFC 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_IMPORT] 0x100 0x0 VirtualAddress: 0xB78C 0x104 0x4 Size: 0x4F>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_RESOURCE] 0x108 0x0 VirtualAddress: 0xC000 0x10C 0x4 Size: 0x608>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_EXCEPTION] 0x110 0x0 VirtualAddress: 0x0 0x114 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_SECURITY] 0x118 0x0 VirtualAddress: 0x0 0x11C 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_BASERELOC] 0x120 0x0 VirtualAddress: 0xE000 0x124 0x4 Size: 0xC>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_DEBUG] 0x128 0x0 VirtualAddress: 0xB720 0x12C 0x4 Size: 0x1C>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_COPYRIGHT] 0x130 0x0 VirtualAddress: 0x0 0x134 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_GLOBALPTR] 0x138 0x0 VirtualAddress: 0x0 0x13C 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_TLS] 0x140 0x0 VirtualAddress: 0x0 0x144 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_LOAD_CONFIG] 0x148 0x0 VirtualAddress: 0x0 0x14C 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_BOUND_IMPORT] 0x150 0x0 VirtualAddress: 0x0 0x154 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_IAT] 0x158 0x0 VirtualAddress: 0x2000 0x15C 0x4 Size: 0x8>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_DELAY_IMPORT] 0x160 0x0 VirtualAddress: 0x0 0x164 0x4 Size: 0x0>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_COM_DESCRIPTOR] 0x168 0x0 VirtualAddress: 0x2008 0x16C 0x4 Size: 0x48>,
 <Structure: [IMAGE_DIRECTORY_ENTRY_RESERVED] 0x170 0x0 VirtualAddress: 0x0 0x174 0x4 Size: 0x0>]
>>>

[recvfrom]

pefile seems to be doing the correct thing -- NumberOfRvaAndSizes can legitimately be less than 16, and in that case the bytes associated with the upper data directories might not be valid. For more info, see: https://stackoverflow.com/questions/43306896/image-optional-header-datadirectory-has-fixed-or-variable-size

Can you give more information on the sample you are analyzing?

[huettenhain]

I have quite the commodity sample here that does what I describe. The file with the following SHA-256 hash is an AgentTesla sample available from VT:

b3b7376c5046be978b5558e91a515c1bf57c13a1151d225745c2bdc3183e0a8f

If has the NumberOfRvaAndSizes set to 0xE, which hides the DATA_DIRECTORY entry for the .NET header. The file executes fine and dnSpy also parses the .NET header correctly, but my pefile based tooling could not, because the corresponding entry is missing from the DATA_DIRECTORY array.

I understand that dealing with malformed executables is a bit of a tough call; this issue & PR are just a suggestion, I can understand if pefile wants to not open this can of worms.

[nightlark]

pefile seems to be doing the correct thing -- NumberOfRvaAndSizes can legitimately be less than 16, and in that case the bytes associated with the upper data directories might not be valid. For more info, see: https://stackoverflow.com/questions/43306896/image-optional-header-datadirectory-has-fixed-or-variable-size

It seems like all implementations of winnt.h (from the Windows 10 SDK, WINE, MinGW) define IMAGE_NUMBEROF_DIRECTORY_ENTRIES as a constant 16, and a cursory look into the .NET runtime and Golang PE decoders are all assuming that there is always space for 16 entries.

I know the various sources of documentation say that NumberOfRvaAndSizes should be used to avoid probing too far, but it is seeming like that differs from how the number of data directory entries is actually implemented (particularly given that there are samples that reduce NumberOfRvaAndSizes but still expect the later entries to be present).

Looking at https://learn.microsoft.com/en-us/windows/win32/debug/pe-format#optional-header-image-only, it mentions using the SizeOfOptionalHeader field to validate that a probe for a data directory does not go beyond the size of the optional header -- I'm wondering if using this header would be a more reliable indication of how much space for data directories is actually present.

[jhhcs]

Notably, LIEF does parse this as I think it should be parsed.