Add climtools uenv #3

leclairm · 2025-02-26T10:30:20Z

It's mainly a clone and rename of netcdf-tools. eccodes is added and netcdf versions are left to the concretizer.

leclairm · 2025-02-26T10:37:52Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

leclairm · 2025-02-26T21:58:25Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

bcumming

A couple of things to think about.

bcumming · 2025-02-27T07:10:33Z

recipes/climtools/25.2/gh200/environments.yaml

+  - cdo
+  - nco
+  - [email protected]
+  - eccodes +tools +aec +openmp jp2k=jasper


Do you need to use the same version of eccodes as was used for the icon/25.2 uenv?
If so, see the icon/25.2 recipe:

it builds a specific version

it contains a repo path with a custom eccodes recipe that makes some tweaks to get the specific version required for the MCH v8 stack.

I saw that. Wanted to ask @jonasjucker, I never use eccodes myself.

I would stick to the same eccodes version as in icon/25.2. Some standardization should be good.

bcumming · 2025-02-27T07:13:21Z

recipes/climtools/25.2/gh200/environments.yaml

+      gpu: false
+  unify: true
+  specs:
+  - hdf5


hdf5 +szip +hl: the hl option is required if anybody wants to use the Python HDF5 interface.

Thanks for the info, I can add it in case. Not sure anybody directly uses hdf5 from python though. People use the netcdf APIs, which should then interface with hdf5 through C. Anyhow, I guess it doesn't harm.

jonasjucker · 2025-02-27T08:17:52Z

recipes/climtools/25.2/gh200/environments.yaml

+  unify: true
+  specs:
+  - hdf5
+  - netcdf-c


I would explicitly list a version fort netcdf-c and netcdf-fortran.
This makes it more transparent and debuggable.

In the end it would be good to share the same versions across Balfrin and Santis

For Balfrin and Santis, yes for sure. We'll eventually deploy both uenvs (ICON and Climtools) there.
I'll check which version the concretizer picked for netcdf-c, netcd-cxx4 and hdf5 and add them explicitly. Re netcdf-fortran, it's not in the uenv, I thought it was useless. What about eccodes? Do we want to keep it in sync with the ICON uenv? I don't know who uses it outside of ICON.

To compile icontools we need netcdf-fortran and eccodes.

I don't understand why you made to effort to add comments with the versions, but removed them from spec.

leclairm · 2025-02-27T10:24:23Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

leclairm · 2025-02-27T12:58:20Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

jonasjucker · 2025-02-27T13:09:59Z

recipes/climtools/25.2/gh200/environments.yaml

+  # - the python minor version is set to the latest possible spack accepts
+  # - other versions where just picked by the spack concretizer
+  - hdf5 +szip +hl  # concretized to @1.14.3
+  - netcdf-c  # concretized to @4.9.2


The comments are confusing, soon this will be a copy-paste artifact that is wrong.

I would put version where we care: netcdf-c, netcdf-fortran and eccodes.
All other spec can live w/o. So also no comments.

Fine, I wasn't 100% convinced myself. Why do we care about the netcdf-c version (there's no netcdf-fortran)? It's just the backend used by python packages or other applications like cdo or nco. Or do you plan to build specific applications with it?

As I stated, we need netcdf-fortran for icontools!

This has to go into this uenv, since it is probably to most prominent code that will use climtools

Since netcdf-fortran is just the interface to netcdf-c I thinnk it is good to know the combination of both.

But please remove the comments stating the version.

bcumming · 2025-02-28T09:08:16Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

leclairm · 2025-03-06T10:45:03Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.3

leclairm · 2025-03-06T10:48:31Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.3

jonasjucker · 2025-03-06T11:58:31Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.3

jonasjucker · 2025-03-06T13:28:44Z

@leclairm There is a syntax error in the script.

leclairm · 2025-03-06T20:28:32Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.3

jonasjucker

Now it looks good. All version pinned.

bcumming · 2025-03-12T12:01:40Z

There is a new PR for an updated version of the icon uenv (#4)

If we want to use the same version of eccodes across all of the uenv (icon, climtools, .... etc), it would be easier to centralise the package definition?

Maybe that is a bit too complicated to do now, but we should at least ensure that the same eccodes package definition is used in all of the 25.2 uenv releases.

bcumming · 2025-03-12T12:03:44Z

We have an icon/25.2 uenv deployed, and this PR and #4 add climtools/25.3 and icon/25.3.
Is the plan to use the 25.3 releases as the "official" releases going forward, and deprecate 25.2?

You probably want to stick to consistent versioning so that the versions of packages in climtools/X and icon/X are consistent, so that users know that they can use the uenv together.

mjaehn · 2025-03-25T14:33:15Z

@leclairm Any status update here?

- use the same spack version as spack-c2sm. This downgrades cdo to 2.4.0 and only allows python 3.12 - unpin package versions, only document what they were concretized to.

Try to use ${GITLAB_DKRZ_TOKEN} through GIT_CONFIG_XXX env vars in the pipeline in order to clone the private icontools repo.

leclairm · 2025-04-03T15:00:16Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

leclairm · 2025-04-04T15:19:11Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

leclairm · 2025-04-04T18:58:38Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

… a branch

bcumming · 2025-06-04T11:03:22Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

bcumming · 2025-06-10T15:50:14Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

bcumming · 2025-06-26T10:09:21Z

cscs-ci run alps;system=santis;uarch=gh200;uenv=climtools:25.2

bcumming · 2025-06-26T10:11:32Z

cscs-ci run alps;system=balfrin;uarch=a100;uenv=climtools:25.2

bcumming · 2025-06-26T10:11:44Z

cscs-ci run alps;system=santis;uarch=zen3;uenv=climtools:25.2

leclairm marked this pull request as ready for review February 26, 2025 22:02

leclairm requested review from bcumming and jonasjucker February 26, 2025 22:02

bcumming requested changes Feb 27, 2025

View reviewed changes

jonasjucker reviewed Feb 27, 2025

View reviewed changes

leclairm requested review from bcumming and jonasjucker March 6, 2025 11:01

jonasjucker approved these changes Mar 11, 2025

View reviewed changes

leclairm force-pushed the climtools branch from ff28118 to c55076e Compare March 26, 2025 11:24

leclairm added 7 commits March 27, 2025 16:25

add: climtools uenv

9b937c1

add: climtools in pipeline

5c993db

add(climtools): hdf5 and netcdf explicitely

15937d9

add: explicit versions + eccodes recipe for 2.36.4

b5b0b78

ref: downgrade spack, unpin versions

0211aec

- use the same spack version as spack-c2sm. This downgrades cdo to 2.4.0 and only allows python 3.12 - unpin package versions, only document what they were concretized to.

add: git-lfs

3605d2b

ref: remove comments about concretized versions

0035c91

leclairm added 4 commits March 27, 2025 16:25

add(climtools): icontools

78484da

Try to use ${GITLAB_DKRZ_TOKEN} through GIT_CONFIG_XXX env vars in the pipeline in order to clone the private icontools repo.

fix: missing CI config for climtools/25.3

8d4231b

test: change CI env var scope

7b80fb5

ref: climtools 25.3 -> 25.2

3850a32

leclairm force-pushed the climtools branch from b157942 to 3850a32 Compare March 27, 2025 15:26

leclairm added 2 commits April 3, 2025 16:56

test: try new CI version with env var

78c5395

fix: roll back to 25.2

0fa9976

fix: spack version that fixes module issues

b1dbf91

env. variable forwarding was merged to uenv-pipeline - no need to use…

f945369

… a branch

Add climtools uenv #3

Are you sure you want to change the base?

Add climtools uenv #3

Uh oh!

Conversation

leclairm commented Feb 26, 2025

Uh oh!

leclairm commented Feb 26, 2025

Uh oh!

leclairm commented Feb 26, 2025

Uh oh!

bcumming left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leclairm Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leclairm commented Feb 27, 2025

Uh oh!

leclairm commented Feb 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leclairm Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bcumming commented Feb 28, 2025

Uh oh!

leclairm commented Mar 6, 2025

Uh oh!

leclairm commented Mar 6, 2025

Uh oh!

jonasjucker commented Mar 6, 2025

Uh oh!

jonasjucker commented Mar 6, 2025

Uh oh!

leclairm commented Mar 6, 2025

Uh oh!

jonasjucker left a comment

Choose a reason for hiding this comment

Uh oh!

bcumming commented Mar 12, 2025

Uh oh!

bcumming commented Mar 12, 2025

Uh oh!

mjaehn commented Mar 25, 2025

Uh oh!

leclairm commented Apr 3, 2025

Uh oh!

leclairm commented Apr 4, 2025

Uh oh!

leclairm commented Apr 4, 2025

Uh oh!

bcumming commented Jun 4, 2025

Uh oh!

bcumming commented Jun 10, 2025

Uh oh!

bcumming commented Jun 26, 2025

leclairm Feb 27, 2025 •

edited

Loading

leclairm Feb 27, 2025 •

edited

Loading