Segfault on building TSAN-enabled Julia #48031

MilesCranmer · 2022-12-28T23:21:12Z

I am unable to build a TSAN-enabled Julia on any recent version of Julia on Ubuntu 20.04 with an x86_64 architecture. However, I can build an ASAN-enabled version; just not a TSAN-enabled version (recommended by @vtjnash for debugging #47957).

I am building inside a dockerfile for reproducibility:

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    cmake \
    git \
    wget \
    ca-certificates \
    curl \
    gpg-agent \
    software-properties-common \
    python3 \
    python3-dev \
    tar \
    xz-utils \
    gfortran

RUN wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh && ./llvm.sh 13

ARG JLVERSION=1.8.3

WORKDIR /downloads

RUN git clone https://github.com/JuliaLang/julia && \
    cd julia && \
    git checkout v${JLVERSION}

# Build:
RUN mkdir -p /workspace && cd julia && contrib/tsan/build.sh /workspace

With this, one builds a TSAN-enabled Julia, version 1.8.4, with docker build --build-arg JLVERSION=1.8.4 -t julia .. On 1.8.3 and 1.8.4, I see the following segfault during the build:

    JULIA /workspace/tsan/usr/lib/julia/corecompiler.ji
Segmentation fault (core dumped)
make[1]: *** [/downloads/julia/sysimage.mk:61: /workspace/tsan/usr/lib/julia/corecompiler.ji] Error 139
make: *** [/downloads/julia/Makefile:82: julia-sysimg-ji] Error 2
The command '/bin/sh -c mkdir -p /workspace && cd julia && contrib/tsan/build.sh /workspace' returned a non-zero code: 2

I am unable to build older versions due to this issue: #43896 (comment).

The text was updated successfully, but these errors were encountered:

MilesCranmer · 2022-12-29T01:54:57Z

cc @vchuravy (looks to have the most recent edit; sorry if this tag is misplaced)

gbaraldi · 2022-12-29T12:38:25Z

I believe @Keno was the last person to play with TSAN.

MilesCranmer · 2023-01-07T19:17:28Z

This is unfortunately really holding me back in my dev work. @Keno were you able to build TSAN-enabled Julia before? Any other tips to get it working?

Keno · 2023-01-07T19:21:31Z

It worked a few months ago when I was doing the MSAN work, but I haven't tried it since. I don't know why it would segfault, somebody would have to go in and debug it.

MilesCranmer · 2023-01-07T19:23:55Z

Do you mean you could build a TSAN one? I am actually able to build ASAN-enabled Julia just fine, it's just the TSAN one that breaks. I'd happy to help debug it because I can't do much else in the meantime; let me know where I should look.

Keno · 2023-01-07T19:24:25Z

Do you mean you could build a TSAN one?

Yes

MilesCranmer · 2023-01-07T19:25:19Z

Cool. Do you remember which Julia version, architecture, and which OS worked for you? I'm happy to use the exact same settings to get something working.

MilesCranmer · 2023-01-10T13:52:41Z

Here are the last 100 lines of my compilation attempt in case this helps. I tried it on 1.9.0-beta2:

    #25 jl_parse_eval_all toplevel.c:? (libjulia-internal-debug.so.1+0xe9c7a)
    #26 ijl_load_ ??:? (libjulia-internal-debug.so.1+0xe93fe)
    #27 ijl_load ??:? (libjulia-internal-debug.so.1+0xe9fdb)
    #28 _finish_julia_init init.c:? (libjulia-internal-debug.so.1+0xa8437)
    #29 julia_init ??:? (libjulia-internal-debug.so.1+0xa7f78)
    #30 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1+0x14f442)
    #31 jl_load_repl ??:? (libjulia-debug.so.1+0x12323)
    #32 main ??:? (julia-debug+0x4e5b3d)

SUMMARY: ThreadSanitizer: signal-unsafe call inside of a signal :? in __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<unsigned long const, objfileentry_t> > >::deallocate(std::_Rb_tree_node<std::pair<unsigned long const, objfileentry_t> >*, unsigned long)
==================
_start at /workspace/tsan/usr/bin/julia-debug (unknown line)
==================
WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=14878)
    #0 free /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:726 (julia-debug+0x462ab0)
    #1 jl_print_native_codeloc :? (libjulia-internal-debug.so.1+0x11dfc8)
    #2 jl_print_bt_entry_codeloc :? (libjulia-internal-debug.so.1+0x11e10a)
    #3 jl_critical_error :? (libjulia-internal-debug.so.1+0x153675)
    #4 sigdie_handler signal-handling.c:? (libjulia-internal-debug.so.1+0x152c7d)
    #5 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2021 (julia-debug+0x4613c8)
    #6 ForwardingMemoryManager::allocateCodeSection(unsigned long, unsigned int, unsigned int, llvm::StringRef) :? (libjulia-codegen-debug.so.1+0x1b877d)
    #7 llvm::RuntimeDyldImpl::emitSection(llvm::object::ObjectFile const&, llvm::object::SectionRef const&, bool) [clone .localalias.14] RuntimeDyld.cpp:? (libLLVM-14jl.so+0x28d5ffd)
    #8 llvm::orc::BasicIRLayerMaterializationUnit::materialize(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >) [clone .localalias.8] Layer.cpp:? (libLLVM-14jl.so+0x2856730)
    #9 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext) jitlayers.cpp:? (libjulia-codegen-debug.so.1+0x17bf25)
    #10 jl_generate_fptr_for_unspecialized_impl ??:? (libjulia-codegen-debug.so.1+0x17ced8)
    #11 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1+0x5a281)
    #12 _jl_invoke gf.c:? (libjulia-internal-debug.so.1+0x5fc20)
    #13 ijl_invoke ??:? (libjulia-internal-debug.so.1+0x5f9ea)
    #14 jl_toplevel_eval_flex :? (libjulia-internal-debug.so.1+0xe6060)
    #15 jl_parse_eval_all toplevel.c:? (libjulia-internal-debug.so.1+0xe9c7a)
    #16 ijl_load_ ??:? (libjulia-internal-debug.so.1+0xe93fe)
    #17 ijl_load ??:? (libjulia-internal-debug.so.1+0xe9fdb)
    #18 _finish_julia_init init.c:? (libjulia-internal-debug.so.1+0xa8437)
    #19 julia_init ??:? (libjulia-internal-debug.so.1+0xa7f78)
    #20 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1+0x14f442)
    #21 jl_load_repl ??:? (libjulia-debug.so.1+0x12323)
    #22 main ??:? (julia-debug+0x4e5b3d)

SUMMARY: ThreadSanitizer: signal-unsafe call inside of a signal :? in jl_print_native_codeloc
==================
==================
WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=14878)
    #0 free /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:726 (julia-debug+0x462ab0)
    #1 jl_print_native_codeloc :? (libjulia-internal-debug.so.1+0x11dfd1)
    #2 jl_print_bt_entry_codeloc :? (libjulia-internal-debug.so.1+0x11e10a)
    #3 jl_critical_error :? (libjulia-internal-debug.so.1+0x153675)
    #4 sigdie_handler signal-handling.c:? (libjulia-internal-debug.so.1+0x152c7d)
    #5 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2021 (julia-debug+0x4613c8)
    #6 ForwardingMemoryManager::allocateCodeSection(unsigned long, unsigned int, unsigned int, llvm::StringRef) :? (libjulia-codegen-debug.so.1+0x1b877d)
    #7 llvm::RuntimeDyldImpl::emitSection(llvm::object::ObjectFile const&, llvm::object::SectionRef const&, bool) [clone .localalias.14] RuntimeDyld.cpp:? (libLLVM-14jl.so+0x28d5ffd)
    #8 llvm::orc::BasicIRLayerMaterializationUnit::materialize(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >) [clone .localalias.8] Layer.cpp:? (libLLVM-14jl.so+0x2856730)
    #9 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext) jitlayers.cpp:? (libjulia-codegen-debug.so.1+0x17bf25)
    #10 jl_generate_fptr_for_unspecialized_impl ??:? (libjulia-codegen-debug.so.1+0x17ced8)
    #11 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1+0x5a281)
    #12 _jl_invoke gf.c:? (libjulia-internal-debug.so.1+0x5fc20)
    #13 ijl_invoke ??:? (libjulia-internal-debug.so.1+0x5f9ea)
    #14 jl_toplevel_eval_flex :? (libjulia-internal-debug.so.1+0xe6060)
    #15 jl_parse_eval_all toplevel.c:? (libjulia-internal-debug.so.1+0xe9c7a)
    #16 ijl_load_ ??:? (libjulia-internal-debug.so.1+0xe93fe)
    #17 ijl_load ??:? (libjulia-internal-debug.so.1+0xe9fdb)
    #18 _finish_julia_init init.c:? (libjulia-internal-debug.so.1+0xa8437)
    #19 julia_init ??:? (libjulia-internal-debug.so.1+0xa7f78)
    #20 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1+0x14f442)
    #21 jl_load_repl ??:? (libjulia-debug.so.1+0x12323)
    #22 main ??:? (julia-debug+0x4e5b3d)

SUMMARY: ThreadSanitizer: signal-unsafe call inside of a signal :? in jl_print_native_codeloc
==================
==================
WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=14878)
    #0 free /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:726 (julia-debug+0x462ab0)
    #1 jl_print_native_codeloc :? (libjulia-internal-debug.so.1+0x11dff6)
    #2 jl_print_bt_entry_codeloc :? (libjulia-internal-debug.so.1+0x11e10a)
    #3 jl_critical_error :? (libjulia-internal-debug.so.1+0x153675)
    #4 sigdie_handler signal-handling.c:? (libjulia-internal-debug.so.1+0x152c7d)
    #5 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2021 (julia-debug+0x4613c8)
    #6 ForwardingMemoryManager::allocateCodeSection(unsigned long, unsigned int, unsigned int, llvm::StringRef) :? (libjulia-codegen-debug.so.1+0x1b877d)
    #7 llvm::RuntimeDyldImpl::emitSection(llvm::object::ObjectFile const&, llvm::object::SectionRef const&, bool) [clone .localalias.14] RuntimeDyld.cpp:? (libLLVM-14jl.so+0x28d5ffd)
    #8 llvm::orc::BasicIRLayerMaterializationUnit::materialize(std::unique_ptr<llvm::orc::MaterializationResponsibility, std::default_delete<llvm::orc::MaterializationResponsibility> >) [clone .localalias.8] Layer.cpp:? (libLLVM-14jl.so+0x2856730)
    #9 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext) jitlayers.cpp:? (libjulia-codegen-debug.so.1+0x17bf25)
    #10 jl_generate_fptr_for_unspecialized_impl ??:? (libjulia-codegen-debug.so.1+0x17ced8)
    #11 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1+0x5a281)
    #12 _jl_invoke gf.c:? (libjulia-internal-debug.so.1+0x5fc20)
    #13 ijl_invoke ??:? (libjulia-internal-debug.so.1+0x5f9ea)
    #14 jl_toplevel_eval_flex :? (libjulia-internal-debug.so.1+0xe6060)
    #15 jl_parse_eval_all toplevel.c:? (libjulia-internal-debug.so.1+0xe9c7a)
    #16 ijl_load_ ??:? (libjulia-internal-debug.so.1+0xe93fe)
    #17 ijl_load ??:? (libjulia-internal-debug.so.1+0xe9fdb)
    #18 _finish_julia_init init.c:? (libjulia-internal-debug.so.1+0xa8437)
    #19 julia_init ??:? (libjulia-internal-debug.so.1+0xa7f78)
    #20 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1+0x14f442)
    #21 jl_load_repl ??:? (libjulia-debug.so.1+0x12323)
    #22 main ??:? (julia-debug+0x4e5b3d)

SUMMARY: ThreadSanitizer: signal-unsafe call inside of a signal :? in jl_print_native_codeloc
==================
Allocations: 0 (Pool: 0; Big: 0); GC: 0
Aborted (core dumped)
make[1]: *** [/downloads/julia/sysimage.mk:61: /workspace/tsan/usr/lib/julia/corecompiler.ji] Error 134
make: *** [/downloads/julia/Makefile:84: julia-sysimg-ji] Error 2

The total compilation output resulted in 398 MB of similar ThreadSanitizer warnings being dumped to stdout.

vtjnash · 2023-01-10T13:58:41Z

The first lines would be where the error happened. I have been meaning to figure out how to make a patch that disables TSAN after failures so it does not print these errors afterwards.

Do you have enough memory? The sanitizers need fairly beefy machines.

MilesCranmer · 2023-01-10T14:21:09Z

Thanks, will get back to you on the memory. The machine has enough (750 GB RAM) but since I'm building inside an insitute-provided docker service, I am asking how much each container has access to.

Thanks for the tip about the threadsanitizer warnings. Here's the filtered output (for my own reference, this is cat nohup.out | vims -n '/Building Julia/,/WARNING:/p' to extract the actual error):

Building Julia...

[removed]

    ^[[32;1mJULIA^[[0m ^[[37;1m/workspace/tsan/usr/lib/julia/corecompiler.ji^[[0m
julia-debug: /downloads/julia/src/cgmemmgr.cpp:865: virtual uint8_t *(anonymous namespace)::RTDyldMemoryManagerJL::allocateCodeSection(uintptr_t, unsigned int, unsigned int, llvm::StringRef): Assertion `!code_allocated' failed.
 
[14878] signal (6.-6): Aborted
in expression starting at boot.jl:243
/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x25
/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x25
/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x25
/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x25
==================
WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=14878)

Any ideas what this is from? Maybe related: https://sourceware.org/bugzilla/show_bug.cgi?id=27521.

vtjnash · 2023-01-10T14:22:46Z

I think we usually build without assertions to bypass that error

MilesCranmer · 2023-01-10T14:30:15Z

I'm using the contrib/tsan/Make.user.tsan without any modifications. Are there any parts of this makefile which would turn on assertions?

TOOLCHAIN=$(BUILDROOT)/../toolchain
BINDIR=$(TOOLCHAIN)/usr/bin
TOOLDIR=$(TOOLCHAIN)/usr/tools

# use our new toolchain
USECLANG=1
override CC=$(TOOLDIR)/clang
override CXX=$(TOOLDIR)/clang++

USE_BINARYBUILDER_LLVM=1

override SANITIZE=1
override SANITIZE_THREAD=1

# default to a debug build for better line number reporting
override JULIA_BUILD_MODE=debug

I also tried adding override FORCE_ASSERTIONS=0 and LLVM_ASSERTIONS=0 to this, but still no luck.

gbaraldi · 2023-01-10T15:19:27Z

Comment out the JULIA_BUILD_MODE=debug. I believe that enforcers assertions

MilesCranmer · 2023-01-10T15:39:12Z

I see, thanks! (I had assumed that was required for the sanitization build). Will give it another go.

MilesCranmer · 2023-01-10T16:19:28Z

Cool, it seems to be building. Thanks for the help! Pushed the fix in #48210.

gbaraldi · 2023-01-10T16:24:39Z

We probably want a separate option that has debug information, but no asserts. Or better yet fix the assertion build but it's nice that at least it's building

MilesCranmer · 2023-01-11T01:15:12Z

~~Still seems to segfault during precompilation [rest of original comment removed]~~

Edit: Nevermind, that binary is produced actually seems to run! So it is working after all.

vtjnash · 2023-01-11T03:43:10Z

It is often built with JULIA_PRECOMPILE=0 for that reason too

MilesCranmer · 2023-01-11T15:19:24Z

Thanks, will try.

By the way, should I file issues for ThreadSanitizer warnings in Julia source code? e.g., things like this (this is Julia 1.8.5):

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=21048)
  Cycle in lock order graph: M0 (0x7b4400000670) => M1 (0x7b44000007b0) => M0

  Mutex M1 acquired here while holding mutex M0 in main thread:
    #0 pthread_mutex_lock <null> (julia-debug+0x6cf5a) (BuildId: 9cd07e78b8f9b5aea494ab1d99a24b598cac1cdd)
    #1 std::mutex::lock() <null> (libLLVM-13jl.so+0x804732)
    #2 JuliaOJIT::findUnmangledSymbol(llvm::StringRef) /downloads/julia/src/jitlayers.cpp:984:12 (libjulia-codegen-debug.so.1+0x172ce4) (BuildId: 145fea905ea03898bad836d4d1843ca40674334e)
    #3 JuliaOJIT::addModule(std::unique_ptr<llvm::Module, std::default_delete<llvm::Module> >) /downloads/julia/src/jitlayers.cpp:952:24 (libjulia-codegen-debug.so.1+0x172775) (BuildId: 145fea905ea03898bad836d4d1843ca40674334e)
    #4 jl_add_to_ee(std::unique_ptr<llvm::Module, std::default_delete<llvm::Module> >) /downloads/julia/src/jitlayers.cpp:1261:25 (libjulia-codegen-debug.so.1+0x16ca7c) (BuildId: 145fea905ea03898bad836d4d1843ca40674334e)
    #5 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long) /downloads/julia/src/jitlayers.cpp:125:13 (libjulia-codegen-debug.so.1+0x16e1e5) (BuildId: 145fea905ea03898bad836d4d1843ca40674334e)
    #6 jl_generate_fptr_for_unspecialized_impl /downloads/julia/src/jitlayers.cpp:380:9 (libjulia-codegen-debug.so.1+0x16f035) (BuildId: 145fea905ea03898bad836d4d1843ca40674334e)
    #7 jl_compile_method_internal /downloads/julia/src/gf.c:2104:13 (libjulia-internal-debug.so.1+0x6d7ea) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #8 _jl_invoke /downloads/julia/src/gf.c:2369:16 (libjulia-internal-debug.so.1+0x72060) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #9 ijl_invoke /downloads/julia/src/gf.c:2384:12 (libjulia-internal-debug.so.1+0x71e2a) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #10 jl_toplevel_eval_flex /downloads/julia/src/toplevel.c:897:18 (libjulia-internal-debug.so.1+0x107280) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #11 jl_parse_eval_all /downloads/julia/src/toplevel.c:1040:22 (libjulia-internal-debug.so.1+0x10ae9a) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #12 ijl_load_ /downloads/julia/src/toplevel.c:1087:26 (libjulia-internal-debug.so.1+0x10a61e) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #13 ijl_load /downloads/julia/src/toplevel.c:1100:26 (libjulia-internal-debug.so.1+0x10b1fb) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #14 _finish_julia_init /downloads/julia/src/init.c:732:9 (libjulia-internal-debug.so.1+0xb9f39) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #15 julia_init /downloads/julia/src/init.c:698:5 (libjulia-internal-debug.so.1+0xb9a6f) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #16 jl_repl_entrypoint /downloads/julia/src/jlapi.c:713:5 (libjulia-internal-debug.so.1+0x16c152) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #17 jl_load_repl /downloads/julia/cli/loader_lib.c:471:12 (libjulia-debug.so.1+0x122d3) (BuildId: 0016c7880e04863086f13589e9fa6c0f7977c649)
    #18 main /downloads/julia/cli/loader_exe.c:59:15 (julia-debug+0xce1bd) (BuildId: 9cd07e78b8f9b5aea494ab1d99a24b598cac1cdd)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M0 acquired here while holding mutex M1 in main thread:
    #0 pthread_mutex_lock <null> (julia-debug+0x6cf5a) (BuildId: 9cd07e78b8f9b5aea494ab1d99a24b598cac1cdd)
    #1 std::mutex::lock() <null> (libLLVM-13jl.so+0x804732)
    #2 jl_add_to_ee(std::unique_ptr<llvm::Module, std::default_delete<llvm::Module> >) /downloads/julia/src/jitlayers.cpp:1261:25 (libjulia-codegen-debug.so.1+0x16ca7c) (BuildId: 145fea905ea03898bad836d4d1843ca40674334e)
    #3 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long) /downloads/julia/src/jitlayers.cpp:125:13 (libjulia-codegen-debug.so.1+0x16e1e5) (BuildId: 145fea905ea03898bad836d4d1843ca40674334e)
    #4 jl_generate_fptr_for_unspecialized_impl /downloads/julia/src/jitlayers.cpp:380:9 (libjulia-codegen-debug.so.1+0x16f035) (BuildId: 145fea905ea03898bad836d4d1843ca40674334e)
    #5 jl_compile_method_internal /downloads/julia/src/gf.c:2104:13 (libjulia-internal-debug.so.1+0x6d7ea) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #6 _jl_invoke /downloads/julia/src/gf.c:2369:16 (libjulia-internal-debug.so.1+0x72060) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #7 ijl_invoke /downloads/julia/src/gf.c:2384:12 (libjulia-internal-debug.so.1+0x71e2a) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #8 jl_toplevel_eval_flex /downloads/julia/src/toplevel.c:897:18 (libjulia-internal-debug.so.1+0x107280) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #9 jl_parse_eval_all /downloads/julia/src/toplevel.c:1040:22 (libjulia-internal-debug.so.1+0x10ae9a) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #10 ijl_load_ /downloads/julia/src/toplevel.c:1087:26 (libjulia-internal-debug.so.1+0x10a61e) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #11 ijl_load /downloads/julia/src/toplevel.c:1100:26 (libjulia-internal-debug.so.1+0x10b1fb) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #12 _finish_julia_init /downloads/julia/src/init.c:732:9 (libjulia-internal-debug.so.1+0xb9f39) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #13 julia_init /downloads/julia/src/init.c:698:5 (libjulia-internal-debug.so.1+0xb9a6f) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #14 jl_repl_entrypoint /downloads/julia/src/jlapi.c:713:5 (libjulia-internal-debug.so.1+0x16c152) (BuildId: f4dcd373c6e811c70674932b5e0e8abae7e3a194)
    #15 jl_load_repl /downloads/julia/cli/loader_lib.c:471:12 (libjulia-debug.so.1+0x122d3) (BuildId: 0016c7880e04863086f13589e9fa6c0f7977c649)
    #16 main /downloads/julia/cli/loader_exe.c:59:15 (julia-debug+0xce1bd) (BuildId: 9cd07e78b8f9b5aea494ab1d99a24b598cac1cdd)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/downloads/julia/usr/bin/julia-debug+0x6cf5a) (BuildId: 9cd07e78b8f9b5aea494ab1d99a24b598cac1cdd) in __interceptor_pthread_mutex_lock

vchuravy · 2023-01-11T15:28:40Z

Yes please. cc: @pchintalapudi

MilesCranmer · 2023-01-11T16:35:04Z

Btw, would it be feasible to have precompiled binaries with TSAN/ASAN enabled? It would help with (1) automatically detecting these issues in Julia, and (2) could be used in CI to trigger failures if sanitization produces warnings in Julia packages. I think it would help improve community health quite a bit, as just running precompilation with TSAN on my machine has highlighted quite a few popular packages out there with lurking sanitization issues. If it was easier to set up TSAN, it would probably be more common to sanitize code.

vchuravy · 2023-01-11T17:28:01Z

We could at least have tsan and asan upload their build artifacts. They pipelines are defined here https://github.com/JuliaCI/julia-buildkite/tree/main/pipelines/main/misc/sanitizers, currently a problem is that tsan doesn't build a system image.

Asan is missing these steps: https://github.com/JuliaCI/julia-buildkite/blob/9c501a475c2d8c1f53fea7fec1fac0537d9d1d4c/utilities/build_julia.sh#L69-L78

Keno · 2023-01-11T19:27:31Z

We did add ABI platform tags for tsan and asan when we added the msan ones, so in theory it's all set, but someone needs to drive pushing it through.

Fixes #48031

brenhinkeller added the building Build system, or building Julia or its dependencies label Dec 29, 2022

MilesCranmer added a commit to MilesCranmer/julia that referenced this issue Jan 10, 2023

Turn off debug build for TSAN, fixes JuliaLang#48031

a412d33

MilesCranmer mentioned this issue Jan 10, 2023

Turn off debug build for TSAN, fixes #48031 #48210

Merged

MilesCranmer mentioned this issue Jan 11, 2023

ThreadSanitizer warnings on Julia 1.8.5 #48231

Closed

vtjnash closed this as completed in #48210 Jan 22, 2023

vtjnash pushed a commit that referenced this issue Jan 22, 2023

build: turn off debug config default for TSAN (#48210)

1ecf73e

Fixes #48031

MilesCranmer mentioned this issue Jan 1, 2024

Upload build artifacts for asan and tsan JuliaCI/julia-buildkite#331

Open

MilesCranmer mentioned this issue Mar 25, 2024

WIP: ImmutableArrays (using EA in Base) #44381

Closed

Uh oh!

Segfault on building TSAN-enabled Julia #48031

Segfault on building TSAN-enabled Julia #48031

Comments

MilesCranmer commented Dec 28, 2022

MilesCranmer commented Dec 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbaraldi commented Dec 29, 2022

Uh oh!

MilesCranmer commented Jan 7, 2023

Uh oh!

Keno commented Jan 7, 2023

Uh oh!

MilesCranmer commented Jan 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Keno commented Jan 7, 2023

Uh oh!

MilesCranmer commented Jan 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MilesCranmer commented Jan 10, 2023

Uh oh!

vtjnash commented Jan 10, 2023

Uh oh!

MilesCranmer commented Jan 10, 2023

Uh oh!

vtjnash commented Jan 10, 2023

Uh oh!

MilesCranmer commented Jan 10, 2023

Uh oh!

gbaraldi commented Jan 10, 2023

Uh oh!

MilesCranmer commented Jan 10, 2023

Uh oh!

MilesCranmer commented Jan 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbaraldi commented Jan 10, 2023

Uh oh!

MilesCranmer commented Jan 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vtjnash commented Jan 11, 2023

Uh oh!

MilesCranmer commented Jan 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vchuravy commented Jan 11, 2023

Uh oh!

MilesCranmer commented Jan 11, 2023

Uh oh!

vchuravy commented Jan 11, 2023

Uh oh!

Keno commented Jan 11, 2023

Uh oh!

MilesCranmer commented Dec 29, 2022 •

edited

Loading

MilesCranmer commented Jan 7, 2023 •

edited

Loading

MilesCranmer commented Jan 7, 2023 •

edited

Loading

MilesCranmer commented Jan 10, 2023 •

edited

Loading

MilesCranmer commented Jan 11, 2023 •

edited

Loading

MilesCranmer commented Jan 11, 2023 •

edited

Loading