clang 17 vs clang 18+ and gcc 13 binaries linking problem

[zlojvavan]

after upgrading ubuntu 23.10 to 24.04 and clang from 17 to 18 cannot build my grpc related projects with clang 18 (and newer) due to link time undefined symbol in debug config:

ld.lld: error: undefined symbol: absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<<<unsigned long, 0>(unsigned long const&)
referenced by log_message.h:132 (/opt/vcpkg/installed/x64-linux/include/absl/log/internal/log_message.h:132)
               bin/server/CMakeFiles/lcs.dir/grpc.cpp.o:(absl::lts_20240116::log_internal::LogMessage::operator<<(unsigned long))

grpc (and abseil) were installed using vcpkg and gcc 13 and I've had no problems building my projects with either clang 17 or gcc 13, but not so with clang 18+

I can see differences in generated symbols between clang 17 and 18, there's missing "U absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <unsigned long, 0>(unsigned long const&)"

~/.vs/LCS/out/build/linux-debug-clang-18/bin/reference/CMakeFiles/lcsref.dir$ nm grpc.cpp.o -C | grep LogMessage::operator 0000000000000000 W absl::lts_20240116::log_internal::LogMessage::operator<<(unsigned long) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <19>(char const (&) [19])

vs

~/.vs/LCS/out/build/linux-debug-clang-17/bin/reference/CMakeFiles/lcsref.dir$ nm grpc.cpp.o -C | grep LogMessage::operator 0000000000000000 W absl::lts_20240116::log_internal::LogMessage::operator<<(unsigned long) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <19>(char const (&) [19]) U absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <unsigned long, 0>(unsigned long const&)

corresponding abseil lib indeed has those templated operator << versions (including <unsigned long, 0>) declared in abseil log_message.h as

 // Types that support `AbslStringify()` are serialized that way.
  template <typename T,
            typename std::enable_if<absl::HasAbslStringify<T>::value,
                                    int>::type = 0>
  LogMessage& operator<<(const T& v) ABSL_ATTRIBUTE_NOINLINE;

  // Types that don't support `AbslStringify()` but do support streaming into a
  // `std::ostream&` are serialized that way.
  template <typename T,
            typename std::enable_if<!absl::HasAbslStringify<T>::value,
                                    int>::type = 0>
  LogMessage& operator<<(const T& v) ABSL_ATTRIBUTE_NOINLINE;

/opt/vcpkg/installed/x64-linux/lib$ nm libabsl_log_internal_message.a -C | grep LogMessage::operator 0000000000000da0 T absl::lts_20240116::log_internal::LogMessage::operator<<(std::ostream& (*)(std::ostream&)) 00000000000000fa t absl::lts_20240116::log_internal::LogMessage::operator<<(std::ostream& (*)(std::ostream&)) [clone .cold] 0000000000000e20 T absl::lts_20240116::log_internal::LogMessage::operator<<(std::ios_base& (*)(std::ios_base&)) 0000000000000120 t absl::lts_20240116::log_internal::LogMessage::operator<<(std::ios_base& (*)(std::ios_base&)) [clone .cold] 00000000000011b0 T absl::lts_20240116::log_internal::LogMessage::operator<<(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) 0000000000001380 T absl::lts_20240116::log_internal::LogMessage::operator<<(std::basic_string_view<char, std::char_traits<char> >) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <signed char, 0>(signed char const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <bool, 0>(bool const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <char, 0>(char const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <double, 0>(double const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <float, 0>(float const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <unsigned char, 0>(unsigned char const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <int, 0>(int const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <unsigned int, 0>(unsigned int const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <long, 0>(long const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <unsigned long, 0>(unsigned long const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <void const*, 0>(void const* const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <void*, 0>(void* const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <short, 0>(short const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <unsigned short, 0>(unsigned short const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <long long, 0>(long long const&) 0000000000000000 W absl::lts_20240116::log_internal::LogMessage& absl::lts_20240116::log_internal::LogMessage::operator<< <unsigned long long, 0>(unsigned long long const&)

so atm I'm unable to build my projects with clang-18+

[DimitryAndric]

My guess is that the templates don't get instantiated, possibly because they are not referenced? At least not in the expected object file? I would be good if you could reduce this quite a bit, since the whole of Abseil is way too much. :)

[zlojvavan]

@DimitryAndric I don't know, it worked as expected with clang-17 and the only difference is used compiler, so something triggered change in behavior in newer clang versions

I was able to build/link all three projects with clang-18 by requesting explicit instantiation in one of common files in my projects:

namespace absl { ABSL_NAMESPACE_BEGIN namespace log_internal { template LogMessage& LogMessage::operator << (unsigned long const&); } ABSL_NAMESPACE_END }

[zlojvavan]

unfortunately despite explicitly specifying "18+" in problem description I was actually unable to test it against clang 19 and 20 as both barks at boost.future but I suppose that's another story:
/opt/vcpkg/installed/x64-linux/include/boost/thread/future.hpp:4671:19: error: no member named 'that' in 'run_it<FutureExecutorContinuationSharedState>'; did you mean 'that_'? 4671 | that_=x.that; | ^ /opt/vcpkg/installed/x64-linux/include/boost/thread/future.hpp:4649:55: note: 'that_' declared here 4649 | shared_ptr<FutureExecutorContinuationSharedState> that_;

[zlojvavan]

I suppose that's another story

already reported

[mering]

We are having the same abseil linking problem when trying to upgrade from LLVM 17 to 18.

I created a reproduction Bazel workspace at https://github.com/mering/llvm-18-linking-issue. See the README.md for instructions.

When changing the llvm_toolchain attribute in MODULE.bazel from 18.1.4 to 17.0.6 this problem disappears and the build succeeds. The different LLVM versions are downloaded from the GitHub releases in this repo (see https://github.com/bazel-contrib/toolchains_llvm/blob/v1.1.2/toolchain/internal/llvm_distributions.bzl).

[DimitryAndric]

I created a reproduction Bazel workspace at https://github.com/mering/llvm-18-linking-issue.

This gives me a 404, probably the access rights not public?

[mering]

I created a reproduction Bazel workspace at https://github.com/mering/llvm-18-linking-issue.

This gives me a 404, probably the access rights not public?

Oops sorry, now it should be public.

[mering]

I played a little with my reproduction example and noticed that it builds correctly with any of the following changes:

• Downgrade LLVM (tested 14.0.0, 16.0.0, 17.0.6)
• Upgrade googletest to 1.15.2
• Upgrade abseil-cpp to 20240722.0 (this also bumps googletest to 1.15.2)
• Remove the --define absl=1 flag for googletest
• Move the gtest.h include below the abseil include
• Explicitly instantiate the template as noted in clang 17 vs clang 18+ and gcc 13 binaries linking problem #102443 (comment)

[mering]

I could reduce the abseil dependency to 3 headers and 1 source file (all with minimal content): https://github.com/mering/llvm-18-linking-issue/tree/main/abseil-cpp/absl.

I verified that it fails to link with 18.1.4 but links just fine with 17.0.6.

Interestingly, when I use absl/strings/has_absl_stringify.h instead of absl/strings/internal/has_absl_stringify.h in test.cc the error disappears. The same change is also present in googletest 1.15.0. This might explain why an upgrade of googletest make the problem disappear as well.

[DimitryAndric]

When I try the bazelisk commands exactly as you specified in that repository, I get an error right away:

$ ./bazelisk-linux-amd64 //test --verbose_failures
Command '//test' not found. Try 'bazel help'.

If I try to use the plain test command, it says there are no tests:

$ ./bazelisk-linux-amd64 test --verbose_failures
INFO: Found 0 test targets...
INFO: Elapsed time: 0.050s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
ERROR: No test targets were found, yet testing was requested

I have roughly zero knowledge of how bazel or bazelisk work, so I can only conclude that it does not work as advertised. :)

[DimitryAndric]

I think it would be much easier if you could just reduce this to a series of compile commands, followed by a link command. I.e. a list of clang -c ... invocations, followed by a clang invocation that links the whole thing together.

Usually this can be scraped from a build tool's log output, when you run it in verbose mode (since hiding what those tools are actually doing is unfortunately the norm these days).

[mering]

You are missing the build command. //test is the target.

The following command should work for you:

./bazelisk-linux-amd64 build //test --verbose_failures

[mering]

If you run with --sandbox_debug, it will print instructions on how you can enter the sandbox. I chose Bazel not only because this is what we use but also to have a reproducible environment with buildroot and toolchain.