ENH: Add IMPACT Similarity Metric to Elastix

[vboussot]

This pull request introduces the IMPACT similarity metric into Elastix, enabling semantic image registration via deep feature representations extracted from TorchScript models.

---

📚 Reference

This metric is described in the following preprint, currently under review:

IMPACT: A Generic Semantic Loss for Multimodal Medical Image Registration
V. Boussot, C. Hémon, J.-C. Nunes, J. Dowling, S. Rouzé, C. Lafond, A. Barateau, J.-L. Dillenseger
arXiv:2503.24121 [cs.CV] – https://arxiv.org/abs/2503.24121

---

⚙️ LibTorch Dependency

IMPACT relies on LibTorch for runtime feature extraction from TorchScript models.

To build Elastix with IMPACT, download and extract LibTorch (e.g., 2.6.0 + CUDA 12.6) and set the Torch_DIR CMake variable when configuring:

 -DTorch_DIR=/path/to/libtorch/share/cmake/Torch

📦 IMPACT Examples & Resources

A full working setup to test the IMPACT similarity metric is available in the following repository:
👉 https://github.com/vboussot/ImpactLoss

This includes:

• 🧠 Pretrained TorchScript models
  Ready-to-use models exported from TotalSegmentator and other large-scale segmentation networks.

• ⚙️ Elastix configuration examples
  Parameter files demonstrating how to use Impact

• 🐳 Docker environment
  A Dockerfile for building Elastix with LibTorch support, preconfigured to run the provided examples.

[N-Dekker]

@vboussot Thank you very much for this great work. I just discussed with Marius (@mstaring) The use of deep features for image registration looks very interesting to us. There is a practical challenge however, how to maintain this new metric. It's a lot of extra code! And I see, it doesn't yet build on the CI, because it depends on Torch and CUDA.

We are able to build your pull request locally on Windows 11, though! Just one code adjustment was necessary: long should be replaced with int64_t! This is necessary, because on Windows (Visual Studio 2022), a long is only 32-bit. It must be int64_t, or order to be compatible with TorchLib's IntArrayRef (which is ArrayRef<int64_t>.) Shall I send you a patch?

Do you think the use of CUDA is essential, or would it also work on the CPU? (I see, https://pytorch.org/get-started/locally/ also offers LibTorch for the CPU.)

Do you think it would be feasible to write a test that would run on the CI, for this pull request?

[vboussot]

Hello @N-Dekker, @mstaring,

Thank you for your interest in the project and for testing the PR on Windows.

I’ve updated the code to use int64_t instead of long to ensure compatibility. However, I can’t confirm it fully resolves all Windows-related issues, feel free to share your patch, as I currently don’t have access to a Windows machine for testing.

The metric now works with the CPU version of LibTorch. However, note that if the Elastix configuration file does not explicitly set GPU = -1, you may encounter the following error: "CUDA is not available. Please check your CUDA installation or run on a compatible device."

As for CI integration, I’ve started setting up a test configuration using LibTorch-CPU. At this stage, the most straightforward and robust approach seems to be configuring Elastix with a user-provided LibTorch version, allowing users to choose between a CPU or CUDA build, depending on their machine and installed CUDA

Valentin

[vboussot]

Hello,

Just a quick update:

The build is now working on both Windows and Ubuntu.
On macOS, LibTorch doesn’t provide a precompiled binary for x86_64, only arm64 is available.
To support x86_64, we’ll need to build LibTorch from source targeting the correct architecture.
I’m planning to add this step to the macOS CI, along with caching to keep build times reasonable.

Regarding the functional test for the IMPACT metric:
I’m currently testing four configurations, across 2D and 3D, in both static and Jacobian modes.
Some tests are timing out, as they perform realistic registration on sample data.
To improve CI duration and consistency, I’ll reduce the number of iterations and switch to deterministic sampling.

Let me know if that sounds good, or if you’d prefer a different setup.

Best,
Valentin

[vboussot]

Thank you @N-Dekker for reporting the bug with such detailed output.

The issue has been resolved in commit 41cebc2. The root cause was a type mismatch in a tensor passed to index_add_(). Specifically, nonZeroJacobianIndices was constructed from a std::vector unsigned long , which led to invalid values (e.g., 4e9, -8e18) on 32-bit platforms when converted to a tensor using torch::from_blob(...) with torch::kLong. An explicit cast to int64_t has been added. This issue only affected index tensors, other types such as double were unaffected. I also ran additional tests in virtualized environments (KVM/QEMU) and didn’t observe any issues.

Finally, the CI has been updated to build LibTorch from source on macOS 13 to support the x86_64 architecture, since official binaries are only available for arm64. The Azure pipeline is now passing successfully. GitHub Actions validation remains to be tested.

[N-Dekker]

The issue has been resolved in commit 41cebc2. The root cause was a type mismatch in a tensor passed to index_add_(). Specifically, nonZeroJacobianIndices was constructed from a std::vector, which led to invalid values (e.g., 4e9, -8e18) on 32-bit platforms when converted to a tensor using torch::from_blob(...) with torch::kLong. An explicit cast to int64_t has been added. This issue only affected index tensors, other types such as double were unaffected. I also ran additional tests in virtualized environments (KVM/QEMU) and didn’t observe any issues.

Very glad to see you fixed this crash so quickly, thanks! I didn't have a clue! But I see now, elastix defines NonZeroJacobianIndicesType = std::vector<unsigned long>:

elastix/Common/Transforms/itkAdvancedTransform.h

Line 133 in cf7567e

using NonZeroJacobianIndicesType = std::vector<unsigned long>;

Maybe we should ban long and unsigned long, both in elastix and ITK, to avoid such problems. 🤷

[vboussot]

The previous version, which used torch::from_blob to directly convert a std::vector into a tensor, was indeed more efficient. However, I chose to use an explicit cast to int64_t instead of modifying NonZeroJacobianIndicesType, to avoid altering elastix internals and risking unintended side effects.

That said, I support the idea of replacing long / unsigned long with fixed-width types like int64_t / uint64_t in both elastix and ITK. This would enhance portability between 32 and 64 bit architectures, with minimal risk of breaking existing functionality.

Regarding the CTest error observed on GitHub Actions. Thanks for approving the tests, I believe it stems from two main issues: First, it's generally more robust to use ${TORCH_LIBRARIES} rather than hardcoding Torch in target_link_libraries(), as CMake targets may not always be correctly exported depending on how LibTorch was built. Second, building PyTorch from source requires Python ≥ 3.9

[N-Dekker]

Regarding PatchSize (default "5*5*5") and VoxelSize (default "1.5*1.5*1.5"), the use of the syntax "i*j*k" is a bit uncommon for elastix parameter values. We would rather have an array of three elements. For example in elastix parameter txt file:

(PatchSize 5 5 5)
(VoxelSize 1.5 1.5 1.5)

Internally, elastix would then store the VoxelSize values as a vector<string> of three elements, being { "1.5", "1.5", "1.5" }.

Would it be possible/feasible to you to adjust these parameters that way? Or would you like me to make it a pull request for https://github.com/vboussot/ImpactElastix/pulls ?

[vboussot]

Thanks for the suggestion, I completely understand the preference for using arrays with space-separated values, as is typically done in elastix.

I had actually considered following the format used for example in FixedImagePyramidRescaleSchedule, but I ended up choosing a string format like "x * y * z" to avoid ambiguity with the multi-resolution syntax, which already uses space-separated values. I also find this format easier to parse consistently for parameters that expect 3D tuples.

That said, I’m of course happy to adapt it if you feel the standard array format would be better for long-term consistency.

[N-Dekker]

I understand your consideration while looking at FixedImagePyramidRescaleSchedule, but I already checked with Marius (@mstaring) and Stefan (@stefanklein), and they also preferred arrays with space-separated values for PatchSize and VoxelSize. Our only concern was that it would break backward compatibility with your experiments, and your paper (the table of hyperparameters). Would that be a problem to you?

[vboussot]

No problem at all, I understand the importance of staying consistent with the rest of elastix, and I’ll make the changes accordingly. It doesn’t cause any issue with the paper, I believe things are clear enough for readers, so it shouldn’t be a problem. I’ll also update the documentation to reflect the new format.