Issues running FDS simulations on Rocky 9

[NhatNguyenEBC]

Hello everyone,

I have an issue with my FDS simulations on the university's cluster. The university cluster was recently updated to Rocky 9. Before the transition to Rocky 9 I did not have the following issue. Since the transition of the cluster to Rocky 9, my simulations begin running but randomly stop without generating any error message and continue until the time limit set in the batch script is reached. This issue occurs sporadically and not with every simulation, making it challenging to identify the cause. The same simulation file runs without issues when restarted in a different folder on the cluster, using identical job settings. The restarted simulation utilized the same total number of cores, however the allocation across nodes was different.

I did not have this issue before the cluster was updated to Rocky 9. Therefore, I assume that the issue is related to the cluster update to Rocky 9. I encountered this issue on the FDS 6.8.0 version (which was compiled on Rocky 8) and the FDS 6.10.1 version (which was compiled on Rocky 9).

Could you please provide guidance or support for resolving this issue? Your assistance would be greatly appreciated. Please let me know if you require additional details about the simulation setup or the cluster. Unfortunately, no error message was generated for the simulations that froze. I have a large number of simulations and limited core-hours avaiable on the cluster, hence I cannot simply run the simulations and restart/repeat the runs that froze repeatedly.

Best Regards

[rmcdermo]

We also ran into this problem with our new cluster, which uses RHEL 9. Two things we noticed.

1. Using the "verbs" fabric for MPI was more stable. The upgrade to Rocky 9 may have kicked you over to a different fabric.
2. When launching the jobs with SLURM it is more stable to specify the number of nodes (so specify --ntasks and --nodes). That is, reserve nodes to match your core count instead of letting SLURM fill in cores where ever it finds available cores.

Since changing to these strategies, we have not seen problems.

p.s. I would also make sure you have installed the latest version of OneAPI from Intel if you are compiling with Intel. If you use gnu, this does not apply.

[NhatNguyenEBC]

1. Using the "verbs" fabric for MPI was more stable.  The upgrade to Rocky 9 may have kicked you over to a different fabric.

The cluster MPI uses a "verb" fabric with (libfabric provider: mlx). And if I am not mistaken the cluster does not offer different verb fabric options.

2. When launching the jobs with SLURM it is more stable to specify the number of nodes (so specify --ntasks and --nodes).  That is, reserve nodes to match your core count instead of letting SLURM fill in cores where ever it finds available cores.

Unfortunately this change did not fix the problem. The simulation still froze with --nodes and --ntasks-per-node.

p.s. I would also make sure you have installed the latest version of OneAPI from Intel if you are compiling with Intel. If you use gnu, this does not apply.

I am using the pre-compiled version of FDS since I am not actively developing FDS. Does it make a difference for the stability whether I use the pre-compiled version of FDS or compile FDS myself?

[mcgratta]

If you are using the precompiled version of FDS for linux, do a simple experiment. At the command prompt on the head or master node, issue this command

mpirun -n 2 -hosts nodename1,nodename2 hostname

where nodename1 and nodename2 are the names of two of the nodes on the cluster. Report what you see when you issue this command.

[marcosvanella]

Try changing srun by mpirun (from the impi library in FDS). srun's behavior is in my experience very much cluster dependent due to slurm setup.

[mcgratta]

This is the script we use for running a job like yours on our cluster, which has 64 cores per node

#!/bin/bash
# /home/mcgratta/firemodels/fds/Utilities/Scripts/qfds.sh -p 100  jobname.fds
#SBATCH -J jobname
#SBATCH -e /home/.../jobname.err
#SBATCH -o /home/.../jobname.log
#SBATCH --partition=batch
#SBATCH --ntasks=100
#SBATCH --cpus-per-task=1
#SBATCH --time=99-99:99:99
#SBATCH --nodes=2
export OMP_NUM_THREADS=1
export I_MPI_DEBUG=5
mpirun  <path to executable>/fds_impi_intel_linux jobname.fds

[NhatNguyenEBC]

My SLURM script setup is essentially the same as yours, except I am using srun instead of mpirun, because my jobs tend to fail due to node errors with mpirun which doesn't happen with srun.

Unfortunately I still experience the freezing error after distributing the cores to a low number of nodes (3 nodes). I attached the corresponding log and slurm file. The simulation froze after about 40 mins while the job ran for 16 hours on the cluster until I cancelled it manually. The same simulation was completed in different runs after about 7 hours.

00_SlurmScript.txt
Cluster_-59860703.log

[mcgratta]

If you post the input file I can try it on our cluster.

[NhatNguyenEBC]

Simulation.txt

This is the input file. However, the issue only occurs sporadically for me and I was not able to reproduce it reliably. Also, similar simulations ran without issue on the Rocky 8 cluster. I encountered the freeze within the simulation when I started running my simulations on Rocky 9.

[mcgratta]

2025_07_14_Ayala2_mid_PL1_TSlicesStuttgart.log
The case ran successfully on our cluster. Attached is the log file.

[amarcozzi]

I'm running into this issue on Scinet Atlas running Rocky 9.x. It doesn't seem to be a reliable error. Sometimes it happens and sometimes it doesn't. When it does happen, it occurs at different points in the simulation. Here are the details of my FDS executable:

(base) anthony.marcozzi@atlas-login-1:~/fds/Build$ ./fds_impi_intel_linux 

 Fire Dynamics Simulator

 Current Date     : August  7, 2025  12:45:47
 Revision         : FDS-6.10.0-19-g275c3be-master
 Revision Date    : Fri Mar 14 14:55:29 2025 -0400
 Compiler         : Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2024.1.0 Build 20240308
 Compilation Date : Mar 17, 2025 19:07:35


 MPI version: 3.1
 MPI library version: Intel(R) MPI Library 2021.12 for Linux* OS

 Hypre library: not used 
 Sundials library: not used
 Consult FDS Users Guide Chapter, Running FDS, for further instructions.

Obviously this is a difficult one as there are all sorts of OS, compiler, slurm, etc. issues that could be leading to this behavior. I wasn't even planning on posting about it until I saw this discussion page.

[mcgratta]

If the exact same job fails at different points for different runs, that probably points to something wrong with the compute cluster. Last year we bought a new cluster with RedHat Enterprise 9. We had some trouble when running many jobs simultaneously, but we could not repeat the errors or determine what was wrong. The only thing I did was change from mlx to verbs for the fabric provider, and that helped. I cannot say why.

[amarcozzi]

Okay thanks for the lead! This is a big help in getting this sorted out. I'll follow up with the HPC managers to see about fixing it on our end, and I'll let you know what we find.