Issue with Frontera spack-stack 1.6.0

[uturuncoglu]

Describe the bug
Under UFS Coastal we are supporting Frontera for the ocean modeling community. We installed 1.6.0 previously without any issue and it was working fine until recent sync with the UFS WM. It seems that fms, g2 and g2tmpl modules are upgraded but still under 1.6.0 and then this requires updating our spack stack installation with those modules.

here are the new module version found in ufs.common
fms/2024.01
g2/3.5.1
g2tmpl/1.13.0

and here are the module that we are using currently,
fms/2023.04.lua
g2/3.4.5.lua
g2tmpl/1.10.2.lua

So, my question is that how this is happening. I was expecting to have release like 1.6.1 if there is some version change in modules. At this point, it is hard for me to update my 1.6.0 branch (has front support) to install these new versions since those new versions are available in newer version of spack and that is not compatible with our spack-stack 1.6.0 branch. Anyway, please let me know if there is 1.6.0 branch in Spack-stack side that has all those updated modules. If not, I think the kind of updates will create more headache in the UFS Coastal side in the future and somehow we need to find a consistent way to handle those kind of issues.

To Reproduce
Steps to reproduce the behavior:

• clone UFS Coastal on Frontera
  git clone --recursive https://github.com/oceanmodeling/ufs-weather-model.git
• try to build one of the configuration
  cd ufs-weather-model/tests
  ./compile.sh frontera "-DAPP=CSTLS -DUSE_ATMOS=ON -DNO_PARAMETIS=OFF -DOLDIO=ON" coastal intel NO NO

Expected behavior
A clear and concise description of what you expected to happen.
expecting to build the configuration

System:
What system(s) are you running the code on?
on Frontera

Additional context
Add any other context about the problem here.
No

[climbfuji]

@uturuncoglu Because some groups are less agile than others (NOAA/UFS, do you hear me?), they do not want to update spack-stack whenever a new release is made. But at the same time, they request updates to specific packages being made available in the existing stack. Bad practice imo, but that's what they insist on. Therefore, when there are no conflicts between packages and modules (especially duplicate module names), we go back and install new versions IN the existing environment OR as an addon/chained environment. The latter is preferred, since it leaves the original environment as-is.

The process generally is like this:

1. Update the local clone of your spack-stack-1.6.0 install from the release/1.6.0 branches upstream (spack submodule and spack-stack parent).
2. Create a chained environment that reuses whatever possible from the original environment and only installs the new packages.
3. Run the install as usual, and add --upstream-modules to the spack module [tcl|lmod] refresh command.

I know this is unsatisfying, and I agree with you that this isn't a good situation. But NOAA&UFS want it this way.

[uturuncoglu]

@climbfuji Thanks for your help. Since UFS WM is used among different applications, it would be nice to follow some kind of standards in dependencies. I am not talking how community will adjust their self to these kind of changes. It is way more complicated for them. At this point, I solved the issue in UFS Coastal side by modifying Frontra module file and not pointing ufs.common in there (I know it is ugly solution but the fatest one). So, I could use the old version of those packages. In the meantime, I could try to follow the same approach like chaining and see what happens. Do you have detailed instruction of it? BTW, the spack that is found in spack-stack 1.6.0 branch is old and does not have newer version of those packages. When I try to update spack submodule, then it was giving error for me and complaining due to I think the inconsistencies between spack-stack and used version of spack. So, it would be time consuming to try installing newer version of those packages using spack-stack 1.6.0 as a base.

[climbfuji]

The "instructions" are here. It's a slightly different approach and requires copying the environment definition from Orion to Frontera: #1180 (comment)

[uturuncoglu]

@climbfuji I circled back to this issue and try to use another system since Frontera was not officially supported when 1.6.0 installed on platforms. So, I switched to derecho to test chaining. At this point, If I checkout 1.6.0 (git clone --recursive -b 1.6.0 https://github.com/JCSDA/spack-stack.git spack-stack-1.6.0) and trying to add new module to it by creating new environment it gives

==> Error: [Errno 2] No such file or directory: '/glade/work/turuncu/COP/spack-stack-1.6.0/spack/share/spack/bootstrap/github-actions-v0.6/metadata.yaml'

Error which seems reasonable since the spack submodule does not have share/spack/bootstrap/github-actions-v0.6 directory. So, it seems that when spack-stack 1.6.0 is installed on Derecho it used different version of Spack which is not shipped with 1.6.0. Anyway, this makes hard to reproduce the environment in my end and add new modules by chaining. Any idea?

For your reference, here are my commands,

git clone --recursive -b 1.6.0 https://github.com/JCSDA/spack-stack.git spack-stack-1.6.0
cd spack-stack-1.6.0
. setup.sh
spack stack create env --name cop --template empty --site derecho --upstream /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.6.0/envs/unified-env/install
cd envs/cop
spack env activate .
spack add conduit@develop+python+adios ^python@3.10.13 ^py-numpy@1.22.3 ^zstd@1.5.2
spack concretize

Any idea or suggestion? Is it possible to confirm that spack-stack 1.6.0 tag is used as it is to install packages on Derecho. If not which version of spack submodule is used?

I also tried to use most recent version of spack-stack but in this case spack is not able to concretize the conduit@develop+python+adios ^python@3.10.13 ^py-numpy@1.22.3 ^zstd@1.5.2 package and give error like cannot satisfy a requirement for package 'python'.

[climbfuji]

Is there a reason you need to work with an old version of spack-stack? It's almost a year old by now. I don't even know where the reference to v0.6 comes from since the 1.6.0 release doesn't know anything about this version (it knows about 0.4 and 0.5). Have you tried to remove ~/.spack?

[uturuncoglu]

@climbfuji Thanks. I could try to remove ~/.spack and see what happens. I think I read something about the chaining that was indicating to use same spack-stack version like upstream but I could not remember where. I also tried to use recent version of spack-stack and as I told you it was giving error like cannot satisfy a requirement for package 'python'. So, I thought that maybe related with the spack-stack or spack version.

Also, I have some confusion about followings

https://github.com/JCSDA/spack-stack/tree/spack-stack-1.6.0 - shows last commit from last year
https://github.com/JCSDA/spack-stack/tree/release/1.6.0 - but this is showing a last commit done in two months

I am to sure but I think the branches are regularly updated due to the requests. Right? Again, I was expecting to freeze everything in each release but it seems it is not the case and it creates lots of confusion.

Anyway, maybe it would be good idea to install entire environment by using same version of packages (with additional ones) rather than chaining. Not sure.

[climbfuji]

The last updates to 1.6.0 are due to the fact that NOAA / UFS were hanging onto this version and skipped 1.7.0 and 1.8.0. They will move to 1.9.0 when it is released. What you are experiencing is the best proof that old versions become unmaintainable and unmanageable.