stripping a binary modified by bolt causes it to crash on startup

[berolinux]

Seen with 18.1.3. Simplest example:

$ cat test.c
#include <stdio.h>
int main(int argc, char **argv) {
  puts("Test");
}
$ clang -o test test.c
$ ./test
Test
$ llvm-bolt -o test.bolt test
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 1b71387311a807d4d600915d1157bf1f9e291e27
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x200000, offset 0x200000
BOLT-INFO: disabling -align-macro-fusion in non-relocation mode
BOLT-INFO: enabling lite mode
BOLT-INFO: 0 out of 9 functions in the binary (0.0%) have non-empty execution profile
BOLT-INFO: removed 2 empty blocks
BOLT-INFO: UCE removed 1 blocks and 7 bytes of code
BOLT: 6 out of 11 functions were overwritten.
BOLT-INFO: patched build-id (flipped last bit)
$ ./test.bolt
Test
$ llvm-strip test.bolt
$ ./test.bolt
Segmentation fault

Same happens when using gcc and BFD strip instead of clang and llvm-strip.

For obvious reasons, stripping a binary before throwing it at bolt is a bad idea -- but there's no equally obvious reason why stripping a binary after running bolt should break it badly.

Also, a bolt-then-strip binary is much bigger than a stripped binary.

[Zentrik]

Striping bolted binaries doesn't work, e.g. #56738

[yota9]

#85796 (comment)

[peterwaller-arm]

I see the original report used llvm-strip from LLVM 18 -- Per #56738 (comment), a fix landed in 19.1; is the issue still present with an LLVM 19 llvm-strip?

[berolinux]

This is partially fixed in 19.1.5.

[bero@ryzen9 ~]$ cat test.c
#include <stdio.h>

int main(int argc, char **argv) {
        puts("Test");
}
[bero@ryzen9 ~]$ clang -o test test.c
[bero@ryzen9 ~]$ ls -l test
-rwxrwxr-x 1 bero bero 6296 Dec  6 16:04 test
[bero@ryzen9 ~]$ llvm-bolt -o test.bolt test
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 26e794f62cf750b57984ae4d74074084ef6af52e
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x200000, offset 0x200000
BOLT-INFO: enabling lite mode
BOLT-INFO: 0 out of 9 functions in the binary (0.0%) have non-empty execution profile
BOLT-INFO: removed 2 empty blocks
BOLT-INFO: UCE removed 1 blocks and 7 bytes of code
BOLT-INFO: patched build-id (flipped last bit)
BOLT: 6 out of 11 functions were overwritten.
[bero@ryzen9 ~]$ ls -l test.bolt
-rwxrwxr-x 1 bero bero 2102336 Dec  6 16:04 test.bolt
[bero@ryzen9 ~]$ ./test.bolt
Test
[bero@ryzen9 ~]$ llvm-strip test.bolt
[bero@ryzen9 ~]$ ls -l test.bolt
-rwxrwxr-x 1 bero bero 2100400 Dec  6 16:04 test.bolt
[bero@ryzen9 ~]$ ./test.bolt
Test
[bero@ryzen9 ~]$ strip test.bolt
strip: stz1qPD4: section .eh_frame lma 0x200340 adjusted to 0x3ff8d8
strip: stz1qPD4: section .eh_frame_hdr lma 0x2003f8 adjusted to 0x3ff990
strip: stz1qPD4: section `.eh_frame' can't be allocated in segment 4
LOAD: .eh_frame .eh_frame_hdr
strip: stz1qPD4: section `.eh_frame_hdr' can't be allocated in segment 4
LOAD: .eh_frame .eh_frame_hdr
[bero@ryzen9 ~]$ ./test.bolt
Segmentation fault (core dumped)

Observations from the above:

• stripping a bolt-ed binary with llvm-strip works now
• it still crashes if it is stripped with strip from GNU binutils instead
• The size issue is still there, even stripped, a 6296 bytes executable turned into a 2.1 MB executable

[paschalis-mpeis]

it still crashes if it is stripped with strip from GNU binutils instead

@berolinux can you please verify that the below patch fixes the issue you mentioned?
https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=bbc969306f8d5fb6c7b636e25f6f8e278946ef23

If so, we could close this issue (similarly to #56738).

Regarding the size issue mentioned, are you sure it is related to bolt+stripping? I would think you'd get some size overhead just with bolt (ie without stripping), it's only with small binaries that it becomes more of a 'dramatic increase'.

[ms178]

@paschalis-mpeis As I've BOLTed some packages lately, I've noticed that GNU binutils issue as well, llvm-strip was fine though. I haven't tested with a newer GNU binutils but will do so once it is released.

Regarding the binary size issue, I've also seen blown up sizes when using certain BOLT options. I heavily suspect --reg-reassign to be the culprit, I've noticed it during the latest python testing and have changed my BOLT options accordingly, see: ms178/archpkgbuilds@68ac4bb

The binary size delta was huge: -145 MB.