Segfault when using `XML` module on MacOS 15.4

[Blacksmoke16]

Bug Report

An oq user reported a crash when using the xml format.

require "xml"

pp XML.parse(%(<name id="1">Fred</name>))

Results in:

Invalid memory access (signal 11) at address 0x20
[0x100e38034] *Exception::CallStack::print_backtrace:Nil +112 in /Users/george/.cache/crystal/crystal-run-test.tmp
[0x100e1cb50] ~procProc(Int32, Pointer(LibC::SiginfoT), Pointer(Void), Nil)@/Users/george/bsmoke/crystal/src/crystal/system/unix/signal.cr:173 +276 in /Users/george/.cache/crystal/crystal-run-test.tmp
[0x189e0f624] _sigtramp +56 in /usr/lib/system/libsystem_platform.dylib
[0x100e4cffc] *GC::free<Pointer(Void)>:Nil +44 in /Users/george/.cache/crystal/crystal-run-test.tmp (2 times)
[0x100e1d448] ~procProc(Pointer(Void), Nil)@/Users/george/bsmoke/crystal/src/xml/libxml2.cr:330 +12 in /Users/george/.cache/crystal/crystal-run-test.tmp
[0x1933b094c] xmlBufFree +92 in /usr/lib/libxml2.2.dylib
[0x1933b08bc] xmlFreeParserInputBuffer +80 in /usr/lib/libxml2.2.dylib
[0x1933b0da4] xmlFreeInputStream +128 in /usr/lib/libxml2.2.dylib
[0x1933b0b10] xmlFreeParserCtxt +44 in /usr/lib/libxml2.2.dylib
[0x1933bc234] xmlDoRead +152 in /usr/lib/libxml2.2.dylib
[0x100ec8d38] *XML::parse<String>:XML::Node +180 in /Users/george/.cache/crystal/crystal-run-test.tmp
[0x100e100dc] __crystal_main +1184 in /Users/george/.cache/crystal/crystal-run-test.tmp
[0x100e83ecc] *Crystal::main_user_code<Int32, Pointer(Pointer(UInt8))>:Nil +12 in /Users/george/.cache/crystal/crystal-run-test.tmp
[0x100e83e18] *Crystal::main<Int32, Pointer(Pointer(UInt8))>:Int32 +64 in /Users/george/.cache/crystal/crystal-run-test.tmp
[0x100e196b0] main +32 in /Users/george/.cache/crystal/crystal-run-test.tmp

Tested with locally built compiler on MacOS 15.4:

$ ccrystal --version
Using compiled compiler at /Users/george/bsmoke/crystal/.build/crystal
Crystal 1.16.0-dev [28479a405] (2025-03-31)

The compiler was not built in release mode.

LLVM: 19.1.7
Default target: aarch64-apple-darwin24.4.0
$ xmllint --version
xmllint: using libxml version 20913

Installing and pointing the linker at the libxml2 homebrew package does work:

$ PKG_CONFIG_PATH="/opt/homebrew/opt/libxml2/lib/pkgconfig" ccrystal test.cr
Using compiled compiler at /Users/george/bsmoke/crystal/.build/crystal
#<XML::Document:0x101362c00 children=[#<XML::Element:0x10136af00 name="name" attributes=[#<XML::Attribute:0x10134ce70 name="id" content="1">] children=[#<XML::Text:0x10136ae00 "Fred">]>]>

So seems like something to do with Mac's bundled system version.

[nwellnhof]

This might be caused by Apple unilaterally removing the custom allocator API of libxml2 in 15.4: https://developer.apple.com/documentation/macos-release-notes/macos-15_4-release-notes#libxml2

[nwellnhof]

In /Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk/usr/include/libxml/globals.h, Apple now has

#if defined(LIBXML_HAS_DEPRECATED_MEMORY_ALLOCATION_FUNCTIONS)
#define xmlMalloc(size) malloc(size)
#define xmlMallocAtomic(size) malloc(size)
#define xmlRealloc(ptr, size) realloc((ptr), (size))
#define xmlMemStrdup(str) strdup(str)
#endif

This doesn't include xmlFree, so using xmlMemSetup or xmlGcMemSetup will result in a mismatch of malloc/free functions and crashes like the one seen above.

[Sija]

@nwellnhof From what you've shown it seems to me this should be reported to Apple bugtracker.

They're getting sloppier by each and every release since some time now 🤦‍♂️

[nwellnhof]

AFAIK, this is a deliberate decision by Apple so they can implement typed allocators. They could have made it degrade in a less mysterious way but in any case, anything using xmlMemSetup is broken with Apple's version of libxml2 since 15.4.

[straight-shoota]

The deprecation notice explicitly asks to stop calling xmlGcMemSetup. So apparently we can't do that anymore on macOS 15.4.

This memory setup has been in place since we started using the library. But I'm not sure why we're even doing that in the first place. LibXML manages its memory explicitly, so it should not need to use the garbage collector.
Apple's intention to use the system allocator should be fine for our use case - although the path to get there seems unnecessarily bumpy.

[nwellnhof]

But I'm not sure why we're even doing that in the first place. LibXML manages its memory explicitly, so it should not need to use the garbage collector.

In a garbage-collected language, the collector should be aware of total memory use. Otherwise, garbage collection could be delayed for too long. See https://samsaffron.com/archive/2018/06/13/ruby-x27-s-external-malloc-problem, for example.

[straight-shoota]

I'm not sure how you could draw such a generic conclusion from that article. As far as I understand, bdw-gc is explicitly meant to be used alongside other allocation methods.

I don't agree on that idea. There should be no need to allocate all memory in the garbage collector. If you want to manage some memory manually, that's totally fine and removes pressure from the collector.
The Crystal runtime certainly doesn't allocate all memory on the garbage collector. Fiber stacks, for example, are memory mapped. Most linked libraries also manage their internal memory using the standard allocator.

[ysbaddaden]

I resonate with the GC having to know the overall memory usage idea —though it mostly only has to deal with its reserved HEAP space— but there are ways to know the actual memory usage of the process (for example rusage).

The other bindings, for example libllvm, libyaml or libpcre2, use the external allocators and call dispose/free methods in #finalize. libxml is the odd one out.

Things to check after moving to a finalizer design, is whether we start getting segfaults or memory leaks.

[straight-shoota]

xmlGcMemSetup is also deprecated in upstream libxml2 in favour of xmlMemSetup, i.e. without atomic malloc.

[straight-shoota]

@ysbaddaden Finalizers should be relatively easy to implement, so we can properly free memory allocated from libxml2.

But we also need to deal with the opposite direction: We're passing pointers of GC-allocated memory to libxml2 (e.g. calling xmlSetProp in XML::Attributes) which the GC wouldn't know about. These allocations could be garbage collected while libxml2 still holds a live pointer.

If we want to uncouple libxml2's memory management from the GC, we need to either

• ensure to keep references to these pointers alive
• copy the data (usually strings) into memory managed by libxml2

[nwellnhof]

I'm not sure how you could draw such a generic conclusion from that article. As far as I understand, bdw-gc is explicitly meant to be used alongside other allocation methods.

The problem with libxml2 is that it can easily create documents consuming 100 MB or more memory. If such an amount of memory was allocated, the GC really must know about it or its decision when to run a collection will be based on skewed data. I can't see how Boehm GC could be different in that regard. Everything will continue work but you'll run GC much too late and waste huge amounts of memory, as explained in the article.

[beta-ziliani]

This is a duplicate of #14174 and #10365 , isn't it? We ought to at least detect the system libxml and error in darwing