how to go from bootstrap (1.5.1) to 1.15.1

[dch]

Bug Report

This is about building latest crystal on FreeBSD, in
the ports & binary pkg system. I'm one of the ports
committers, making this work for all users of FreeBSD.

• FreeBSD builds everything from sources "because"
• ✅ thus we need to bootstrap from LLVM14 -> Crystal 1.5.1
• ✅ then we can build successfully LLVM16 -> Crystal 1.10.1
• ✅ then we can build successfully LLVM20 -> Crystal 1.15.1

But skipping the intermediary Crystal 1.10.1 fails, i.e.
🤯 trying to go directly from bootstrap -> LLVM20 -> Crystal 1.15.1 fails:

===>  Building for crystal-1.15.1
Using /usr/local/bin/llvm-config19 [version=19.1.7]
CRYSTAL_CONFIG_BUILD_COMMIT="84f389ac5424" CRYSTAL_CONFIG_PATH=lib:/usr/local/lib/crystal SOURCE_DATE_EPOCH="1738627200"  CRYSTAL_CONFIG_LIBRARY_PATH='$ORIGIN/../lib/crystal' ./bin/crystal build --progress --stats --threads 16 --release --no-debug -D strict_multi_assign -D preview_overload_order --link-flags="  -fstack-protector-strong "  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib -D use_pcre2
Using compiled compiler at .build/crystal
Parse:                             00:00:00.000026540 (   1.08MB)
Semantic (top level):              00:00:00.457428870 ( 186.13MB)
Semantic (new):                    00:00:00.002755877 ( 186.13MB)
Semantic (type declarations):      00:00:00.054514598 ( 194.13MB)
Semantic (abstract def check):     00:00:00.018375190 ( 218.13MB)
Semantic (restrictions augmenter): 00:00:00.020660960 ( 218.13MB)
Showing last frame. Use --error-trace for full trace.

There was a problem expanding macro 'macro_35976034720'

Code in src/gc/boehm.cr:428:3

 428 | {% if flag?(:preview_mt) %}
       ^
Called macro defined in src/gc/boehm.cr:428:3

 428 | {% if flag?(:preview_mt) %}

Which expanded to:

 > 2 | # support for legacy gc releases
 > 3 | def self.set_stackbottom(stack_bottom : Void*)
 > 4 |   LibGC.stackbottom = stack_bottom
               ^----------
Error: undefined fun 'stackbottom=' for LibGC

From my perspective its better to only need 1 bootstrap step rather than 2.

Is the above bug fixable?

Or should I go the less satisfying but workable multi-stage bootstrap via Crystal 1.10.1?

thanks!

[straight-shoota]

This seems quite strange.
The conditions for generating this macro code should ensure that LibGC.stackbottom is present.
There might be some hickups with version recognition on FreeBSD when pkg-config is missing, but that still shouldn't break in that way.

Also, I'm at a loss what change between 1.5.1 and 1.10.1 would make a difference here.

[ysbaddaden]

Maybe something in 1.11.0? For example #14130

[straight-shoota]

Yes, that could be a relevant change.
But I don't see what difference it would make whether the host compiler is 1.5 or 1.10.

[straight-shoota]

The lib bindings definitely have an error on FreeBSD. We're using pkg-config to discover the lib version, but we're using the wrong name for FreeBSD.

crystal/src/gc/boehm.cr

Line 53 in 942ad79

VERSION = {{ `pkg-config bdw-gc --silence-errors --modversion || printf "0.0.0"`.chomp.stringify }}

But this shouldn't be a bigger issue, and doesn't explain the observed failure. It just means that we're assuming the lowest lib version for the bindings.

[straight-shoota]

The problem here seems to be a misalignment between the lib bindings and their usage. But I don't see how this could happen.

These are the relevant code pieces, simplified:

lib LibGC
  {% if some_condition %}
    fun set_stackbottom = GC_set_stackbottom(th : ThreadHandle, sb : StackBase*) : ThreadHandle
  {% else %}
    $stackbottom = GC_stackbottom : Void*
  {% end %}
end

{% if LibGC.has_method?(:set_stackbottom) %}
  # use LibGC.set_stackbottom
{% else %}
  def set_stackbottom(stack_bottom : Void*)
    LibGC.stackbottom = stack_bottom
  end
{% end %}

In the lib bindings, we conditionally define either fun set_stackbottom or $stackbottom (they have the same purpose). Later on, the code uses either of them, and it's guarded by a condition to check which one is defined. One of them has to.

[ysbaddaden]

I guess the change isn't the sole issue. Something changed between 1.5 and 1.10 that made the change in 1.11 to work with the latter but not the former? Maybe something related to macros?

[ysbaddaden]

@dch We fixed the pkg-config name for Boehm GC in FreeBSD in 6d75bf0 to be released with 1.16.0. Does it have any impact? I guess not, it's something related to macros, but we can try 🤷

[straight-shoota]

I found a lead: Up until 1.7, lib funs were not visible in top-level macros.

In the following example, has_method? returns false in Crystal 1.6 and below:

lib LibFoo
  fun foo : Void
end

{{ LibFoo.has_method?(:foo) }}

This was fixed in #12848 (1.7.0).

So apparently we're actually generating bindings for GC_set_stackbottom, but later fail to recognize that.
I'm not sure how we end up in the branch for GC_set_stackbottom because neither flags preview_mt nor win32 are set and I should expect VERSION to be 0.0.0 unless pkg-config bdw-gc --silence-errors --modversion does actually print an actual version number.

A simple workaround should be to replace the has_method? condition with the original one.

--- i/src/gc/boehm.cr
+++ w/src/gc/boehm.cr
@@ -431,7 +431,7 @@ module GC
       sb.mem_base = stack_bottom
       LibGC.set_stackbottom(thread_handle, pointerof(sb))
     end
-  {% elsif LibGC.has_method?(:set_stackbottom) %}
+  {% elsif flag?(:preview_mt) || flag?(:win32) || compare_versions(VERSION, "8.2.0") >= 0 %}
     # this is necessary because Boehm GC does _not_ use `GC_stackbottom` on
     # Windows when pushing all threads' stacks; it also started crashing on
     # Linux with libgc after v8.2.4; instead `GC_set_stackbottom` must be use

[straight-shoota]

Alternatively, we can move the condition inside the def, at which point everything should work as expected:

--- i/src/gc/boehm.cr
+++ w/src/gc/boehm.cr
@@ -431,21 +431,21 @@ module GC
       sb.mem_base = stack_bottom
       LibGC.set_stackbottom(thread_handle, pointerof(sb))
     end
-  {% elsif LibGC.has_method?(:set_stackbottom) %}
-    # this is necessary because Boehm GC does _not_ use `GC_stackbottom` on
-    # Windows when pushing all threads' stacks; it also started crashing on
-    # Linux with libgc after v8.2.4; instead `GC_set_stackbottom` must be used
-    # to associate the new bottom with the running thread
-    def self.set_stackbottom(stack_bottom : Void*)
-      sb = LibGC::StackBase.new
-      sb.mem_base = stack_bottom
-      # `nil` represents the current thread (i.e. the only one)
-      LibGC.set_stackbottom(nil, pointerof(sb))
-    end
   {% else %}
-    # support for legacy gc releases
     def self.set_stackbottom(stack_bottom : Void*)
-      LibGC.stackbottom = stack_bottom
+      \{% if LibGC.has_method?(:set_stackbottom) %}
+        # this is necessary because Boehm GC does _not_ use `GC_stackbottom` on
+        # Windows when pushing all threads' stacks; it also started crashing on
+        # Linux with libgc after v8.2.4; instead `GC_set_stackbottom` must be used
+        # to associate the new bottom with the running thread
+        sb = LibGC::StackBase.new
+        sb.mem_base = stack_bottom
+        # `nil` represents the current thread (i.e. the only one)
+        LibGC.set_stackbottom(nil, pointerof(sb))
+      \{% else %}
+        # support for legacy gc releases
+        LibGC.stackbottom = stack_bottom
+      \{% end %}
     end
   {% end %}

[ysbaddaden]

Let's go with that!

[ysbaddaden]

Hum, are you sure it does? Isn't it still resolved with the outer if/else macro? i.e. need to escape the inner macros.

[straight-shoota]

Yes, you're right. Added macro escapes.

[straight-shoota]

I got wondering why this issue hasn't appeared before. It's a pretty obvious bug and should've broken down more builds. We're explicitly testing compatibility with old compiler releases in CI, but this has never come up. It looks like it should, though.

This code would always break with a base compiler of 1.6 or below when the condition flag?(:preview_mt) || flag?(:win32) || compare_versions(VERSION, "8.2.0") >= 0 is met. We do not test backward compatibility of preview_mt because it's a preview feature. And we only test backwards compatibility on Linux because breaking issues are usually not system-specific.
The version of libgc should surely be 8.2.0 or above. And yes it actually is, but pkgconfig has no information about it. On Linux, we're shipping libgc as a static library with the compiler (see crystal-lang/distribution-scripts#285).
That means we always end up in the else branch, using the global variable LibGC.stackbottom.

And indeed the issue reproduces with a base compiler 1.6.2 if we have libgc >= 8.2 installed.

> # within container crystallang/crystal:1.6.2-alpine
> apk add gc-dev
(1/3) Installing gc (8.2.0-r0)
(2/3) Installing libgc++ (8.2.0-r0)
(3/3) Installing gc-dev (8.2.0-r0)
OK: 169 MiB in 55 package
> bin/crystal eval 'puts "Hello GC"'
Showing last frame. Use --error-trace for full trace.

There was a problem expanding macro 'macro_140044620167360'

Code in src/gc/boehm.cr:428:3

 428 | {% if flag?(:preview_mt) %}
       ^
Called macro defined in src/gc/boehm.cr:428:3

 428 | {% if flag?(:preview_mt) %}

Which expanded to:

 > 3 |
 > 4 |         # support for legacy gc releases
 > 5 |         LibGC.stackbottom = stack_bottom
                     ^----------
Error: undefined fun 'stackbottom=' for LibGC

Note: More modern docker images have gc-dev installed and libgc.a removed.