[pkg] Build OxCaml with Dune package management

[maiste]

This issue will serve as a meta issue to describe the changes required to build OxCaml with Dune package management.

Currently, I have trying to build it using @gridbugs small repository, hello-oxcaml. The locking happens correctly, but the build encounters various errors:

1. Building the init-compiler requires it to be added to toolchains supported list. Otherwise, as it is not relocatable, it will suffer the same problem as the compiler.
2. The init-*.pkg use the %{prefix} variable, which is causing a problem with Dune Package Management because it points to the _build/.sanbox/... path. We extend the path with the dependencies instead of having a global state as opam would do. The semantic of %{prefix} is not really clear for me in this context. I'm not confident it has one.
3. If we change the path using %{pkg:init-compiler:share} in the init-*.pkg lock files, we are able to build it to a findable path. However, when doing the variable expansion in other *.pkg files (which updates the PATH accordingly), it is not using the path within toolchains but the one within the _build directory. I'm writing a test to reproduce this error.
4. In the final stage, when building the ocaml-variants package (after building init-* packages), it fails because of the problem discovered in dune-pkg: Local commands (ie ./configure) should be picked up from the sandbox rather than the source #11514. Indeed, (run autoconf) generates a configure but it can't be executed.

I have been discussing with @rgrinberg on how to make Dune Package Management worked with OxCaml and more broadly on how to generalize it to support non-relocatable packages.

I'll update the issue to point to the various resources and changes.

[dra27]

The opam packaging has been designed to work well for opam - in particular to allow OxCaml to be updated without having to rebuild all the constituent parts.

The obvious Dune fit would be to building OxCaml in a separate build context (i.e. compiling the Dune stage of OxCaml in a build context which has the correct compiler). I'm wondering (= quite strongly recommending) not adding features to dune pkg just to support the init-* packages, but rather how about looking at refactoring the packages themselves so that OxCaml is installable as a single package in the first instance?

[dra27]

Just as a proof-of-concept for simplifying:

--- a/dune-workspace
+++ b/dune-workspace
@@ -2,7 +2,7 @@

 (repository
  (name jst)
- (url git+https://github.com/janestreet/opam-repository#with-extensions))
+ (url git+https://github.com/dra27/opam-repository#with-extensions-dp))

 (lock_dir
  (repositories upstream overlay jst))

and then:

$ dune --version
"Dune Developer Preview: build 2025-04-26, git revision
1a0d3646a365210cb42a4366cb3a026b4c8928e3"
$ dune pkg lock
$ time dune build
 Downloading ocaml-variants.5.2.0+flambda2
    Building ocaml-variants.5.2.0+flambda2

real    12m59.186s

The first build is taking an inordinately long time¹, but the caching is definitely working after that:

$ git clean -dfX
$ time dune build
    Building ocaml-variants.5.2.0+flambda2

real    0m1.483s

Footnotes

1. I haven't figured out what's going on, but my hunch is it's a failure to detect the appropriate parallelism in the "internal" Dune call in OxCaml's build: forcing a higher -j brought the first-time build down to the ~6minutes that an opam switch switch with the same package was taking. ↩

[gridbugs]

+1 to modifying the package metadata rather than modifying dune. Thanks for the proof-of-concept @dra27!

[gridbugs]

I've updated my example project to use @dra27's fork of the JST repo which internalizes the "init-*" packages and changed the example program to use the new oxcaml syntax to demonstrate that it's being built with the oxcaml compiler.

[maiste]

@dra27 thanks for the proof-of-concept! I agree with your point. Note that some problems we encountered here are related to non-relocatable packages in general. This isn't exclusive to OxCaml. We need to fix these problems anyway for Dune Package Management.

As @gridbugs, I have tried the solution, and I'm able to make it build and run the code!

[maiste]

Some feedback about it, I have been able to push a working version using @dra27 patches and using a custom repository.
It can be found at https://github.com/gridbugs/hello-oxcaml/tree/with-custom-patch.