add nanobind-abi metapackage

[minrk]

pattern copied from pybind11-abi

Based on reading the compatibility notes in the docs, I think this is required just like it is for pybind11.

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

[minrk]

@conda-forge-admin please rerender

[wjakob]

Can you explain how this ABI package works, and how it would be used? Ideally, such explanations are also documented somewhere (the linked pybind11 page doesn't actually explain this part as far as I can tell).

[minrk]

A simpler version of this is to add to nanobind:

run_exports:
  weak_constrains:
    - {{ pin_subpackage("nanobind", max_pin="x.x") }}

which assumes that nanobind may bump the ABI version in a minor release, but will not in a patch release.

Advantage of ABI metapackage: As ABI stabilizes, fewer migrations need to happen
Advantage of minor patch pin: simpler recipe, fewer packages; but each minor release must trigger rebuilds even if ABI hasn't changed

[minrk]

Yes, the most complex case I can think of (and the case I deal with) is fenics, which uses nanobind in 3 environments:

• basix (a dependency of dolfinx)
• dolfinx
• dolfinx code generation at runtime

They have just switched from pybind11 to nanobind. With pybind11, at least, the ABI version must match in all 3 environments in order for compilation to succeed (pybind11 actually required compiler major versions to match exactly, I'm not sure if nanobind does as well). The main goal is to make sure that an incompatible nanobind is not accepted at any point. Since the ABI version is tracked separately, a metapackage to track this works pretty well and allows explicit tracking of what's needed, in order to meet the requirements specified in the nanobind docs.

As I mentioned above, a perhaps simpler approach is to use only the API version and preemptively declare minor versions incompatible because they may bump the ABI. This has little downside if the ABI changes frequently, but can get unnecessarily tedious if the ABI stabilizes significantly relative to minor versions, because every minor release must trigger cascading rebuilds even if they would have been compatible.

[minrk]

conda-forge/fenics-basix-feedstock#17 is a PR switching from pybind11-abi to nanobind.

The key part is:

• run_constrained pinning nanobind at runtime to the same minor version at build time. This is what would have been produced by the weak_constrains suggested above.

[wjakob]

I think it's better not to pin ABI with minor releases, they capture different ideas. It is not inconceivable that I might have to do a patch release with just an ABI version bump, in case that this fixes ABI breakage that accidentally slipped through.

If there is an ABI metapackage, what would be involved when making a new release?

Imagine you are talking to someone who has no idea about how conda and conda-forge work.

[minrk]

If there is an ABI metapackage, what would be involved when making a new release?

To make a new release of nanobind, usually the conda-forge bot will make its automated PR (or you can do it yourself). A version bump makes the following changes to meta.yaml:

• update the version number at the top
• update the hash in build.source
• reset the build number to 0

If the abi version has not changed, this is all that's needed. Merge the PR and you're done.

Adding the abi metapackage adds one additional step:

• update the abi_version at the top of meta.yaml, if changed.

The PR from the conda-forge bot won't do this part, so you'll get a PR with the first 3 steps done. If the abi has changed, the build should fail this check in the abi package tests. This is good, because it prevents you from publishing a release with the wrong abi version reported. You will then need to update the abi version at the top of meta.yaml to match the version in nb_internals.cpp. Conversely, if the build succeeds, you know the abi version has not changed.

(aside: it would be nice if the abi version were also available from Python as e.g. nanobind.abi_version)

[minrk]

Can you explain how this ABI package works, and how it would be used?

I realized I didn't directly answer this.

The nanobind-abi package is an empty metapackage, indicating the abi version of nanobind. It doesn't do or mean anything on its own. The main thing it does is give us something to be versioned separately from the nanobind package version.

The nanobind package has a run_constrained dependency on the matching nanobind-abi, meaning that if nanobind-abi is present in an environment, the nanobind package must be compatible with that abi version.

The nanobind-abi package has run_exports to ensure that whatever nanobind-abi appears in host dependencies, the same version will be strictly pinned in the run dependencies, which thanks to the run_constrained in nanobind means that only an ABI-compatible nanobind can be installed in the environment.

That's the mechanics of it. How it's used:

If a package has dependencies:

name: mypackage
requirements:
  host:
    - nanobind
    - nanobind-abi

it will have runtime dependency on nanobind-abi ==14. Meaning any other package that depends on mypackage and also has nanobind, e.g.

name: downstream
requirements:
  host:
    - mypackage
    - nanobind

will always build with a version of nanobind that is ABI-compatible with the one used to build mypackage, even if there's a more recent version of nanobind available.

This behavior is opt-in for packages by placing nanobind-abi in the host dependencies (i.e. if packages only depend on nanobind they won't have this restriction), which should be done by any package that expects to be used via nanobind, as is done in fenics, I believe.

[wjakob]

Sorry for dropping the ball on this. I read through your helpful explanations, and this now makes a lot of sense. I will merge the PR if you could rebase it. The version and ABI version have changed in the meantime (to 2.2.0 and 15, respectively).

[minrk]

Sure, I can do that. I've also run into a related issue recently, where the major version of the CXX compiler is also part of the ABI of any package built with nanobind (I'm not sure why). So as a result, fenics-dolfin built with gcc 12 cannot run with fenics-basix (its dependency) built with gcc 13, resulting in errors like:

Traceback (most recent call last):
  File "/home/mnotazio/mambaforge/envs/test_env/lib/python3.11/site-packages/dolfinx/fem/element.py", line 91, in _
    return CoordinateElement(_cpp.fem.CoordinateElement_float32(e._e))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: __init__(): incompatible function arguments. The following argument types are supported:
    1. __init__(self, celltype: dolfinx.cpp.mesh.CellType, degree: int) -> None
    2. __init__(self, element: basix::FiniteElement<float>) -> None
    3. __init__(self, celltype: dolfinx.cpp.mesh.CellType, degree: int, variant: int) -> None

Invoked with types: dolfinx.cpp.fem.CoordinateElement_float32, basix._basixcpp.FiniteElement_float64

I don't have a great answer for that, but one option is to make this recipe no longer noarch (the nanobind package can still be noarch), but have the ABI package track the cxx compiler version as well somehow.

[minrk]

Updated to curent version

[MatthiasKohl]

@wjakob I think we can merge this, right?

[MatthiasKohl]

LGTM

[MatthiasKohl]

@wjakob I ended up reading through this, and I agree with it, so I've just merged it

[hmaarrfk]

we recently added some safety checks, you need to add an extra output

[hmaarrfk]

You must do something like this to add a new package: conda-forge/admin-requests#1179