Recommendation for pinning & dependency mixing between sub-packages

[phinate]

What maintenance does this project need?

Hi! A very confused user here. I see that this is a monorepo of three packages that seem to dynamically get versioned based on the relevant github releases. I assume this is done to keep the packages in lock-step since they're tightly coupled.

We're trying to work out how to pin dependencies to ensure things don't break, but we're struggling with that. Due to anemoi-training depending on anemoi-graphs and anemoi-models, but not pinning them or upper-bounding them, if you pin just training, things don't work, since the sub-packages are allowed to float within their lower-bounded requirements. Also, if we pin everything, we're unsure how to identify which version of each sub-package to pin, since they're dynamically tagged with scm based on releases.

What should we do here? I think we expected PyPI to have a package called anemoi-core that installed particular versions of all three packages that are guaranteed to work, but we're left with some interface differences between sub-packages if we pin any permutation of graphs, models or training.

Advice or signposting on how we should track this (and how to robustly grab compatible versions for CI testing) would be great. Thanks!

Organisation

The Alan Turing Institute / UK Met Office

[mchantry]

Hello
Thanks for the first issue! Sorry if this has been frustrating.
We very recently move to releasing all three packages on anemoi core in sync, to give a clearer message about some package compatibility. This can give a clear example of a set of dependencies but does require browsing https://github.com/ecmwf/anemoi-core/releases
An example, the latest version of each package will be a compatible set.

Your idea of a pyproject toml pinning these together is an interesting one to consider. Probably best as automatic process, but this could be possibly integrated into the release workflow. Would you be interested to contribute to this?

We are currently investing in expanded integration testing between the different packages, once this is completed we are considering a compatibility matrix which could be automatically generated by running these tests with the different versions of the packages.
Thanks
Mat