Help needed for case recovery after numerical instability

[FireResearch-BK]

I'm trying to recover a case running with FDS 6.10.1 that crashed on me due to numerical instability shortly after a restart point. Since there was a density warning before, the error is pressure solver related. (I can not remember if it was minimum or maximum density as I've already closed the console window and these warnings are not printed to the .out file.)

The case (which I can not post in public) was initially run with the FFT pressure solver. This is its original header:

&HEAD CHID='[...]', TITLE='[...]'/
&TIME T_END=1800.0/
&DUMP COLUMN_DUMP_LIMIT=.TRUE., DT_DEVC=10.0, DT_HRR=10.0, DT_RESTART=50.0, DT_SLCF=10.0/
&MISC MINIMUM_ZONE_VOLUME=2.0, RESTART=F/
&PRES VELOCITY_TOLERANCE=0.005, MAX_PRESSURE_ITERATIONS=50, SUSPEND_PRESSURE_ITERATIONS=T/

In the past, a crashed case had a good chance to be completed by switching to the ULMAT pressure solver and continuing from a restart point, but this does not appear to work for this one. I've edited the header of the FDS to disable the FFT solver configuration and enable the ULMAT solver instead:

VELOCITY_TOLERANCE=0.005, MAX_PRESSURE_ITERATIONS=50, SUSPEND_PRESSURE_ITERATIONS=T/
&PRES SOLVER='ULMAT'/

Now when I start the case from the restart point at around t = 900 s, it still crashes a few time steps later:

Job TITLE : [...]
ob ID string : [...]

Time Step: 37700, Simulation Time: 900.55527 s
Time Step: 37800, Simulation Time: 901.89588 s
Time Step: 37840, Simulation Time: 902.36698 s

ERROR(374): Numerical Instability - FDS stopped (CHID: [...])

In the .out file, the pressure and velocity errors shoot up to ludicrous values for the time step before the crash:

  Time Step    37840   August 11, 2025  11:08:02
  Step Size:  0.962E-02 s, Total Time:  902.36698 s
  Pressure Iterations: 10
  Maximum Velocity Error:  0.58E+07 on Mesh 10 at (65,10,20)
  Maximum Pressure Error:  0.62E+10 on Mesh 11 at (4,9,10)

The time step before the one with the crash had errors of 0.72E-01 for velocity and 0.12E+03 for pressure.

No mesh exceeds 1 for the CFL number.

I've already tried to decrease the minimum density by adding &CLIP MAXIMUM_DENSITY=2.000, MINIMUM_DENSITY=0.005/ to the input file header, but it still crashes at the same point.

Is there anything else I can try to recover the case?

As there are more than 1,000,000 cells (2,304,649, to be exact, split over 16 meshes, all at the same dx,dy,dz), switching to GLMAT and UGLMAT is not an option.

[FireResearch-BK]

&PRES SOLVER='UGLMAT HYPRE'/ managed to push the case past the previous point of crashing, but in turn crashes with forrtl: severe (157): Program Exception - access violation (symptom of running out of memory?) a few time steps later.

Note that the problematic case is just one of three scenarios, of which the other two ran fine. So the root cause is definitely more of an unfortunate flow occurrence somewhere in the domain than a principal problem with the input file.

[johodges]

Can you post your out, steps, and hrr files?

[FireResearch-BK]

Here's the .out, steps and hrr file. The .out file was trimmed behind misc. parameter information and down to the last 10 logged steps of the original crash.

steps.csv
hrr.csv
out.txt

Before somebody raises an eyebrow due to the ratio of HRR to cell size: The mesh resolution is predermined by standardization and we can't alter it. But as I've said, the dx,dy and dz values normally work fine for the application. It's just this case that's problematic.

[johodges]

I did not see anything obvious in those files. It will be hard to debug without more information on the case.

[FireResearch-BK]

No problem.

The simulation itself had ran long enough to deliver usable results before it crahed. I just wanted to have it run longer to ensure there's nothing else of relevance happening.

[johodges]

One thing you could try is setting DT_UVW, DT_SPEC, and DT_TMP = 900.1 seconds in the file and then restart. Assuming FDS lets you add those on a restart, you will get full field outputs for the flow field. You could then use these to start a "new" simulation which is initialized with those files and set T_BEGIN = 900 seconds. Also, since it is a new simulation, you can make whatever changes to the domain required. This is not perfect as it will not preserve the pre-heating of the solids in the domain. But it would be interesting to see if you still become unstable quickly.

[FireResearch-BK]

As only the smoke spread in the domain is relevant for the simulation, the loss of any preheating would be acceptable (I think).

I might try your suggestion if I get another crash and see something in the results that would warrant having the simulation run some more.

But thanks for the tip. This might be useful for others in a simular situation.

[mcgratta]

Keep in mind that it is not always a pressure problem that causes instabilities, and the diagnostics at the last time step are almost always off the charts because the velocity has gone off the charts. I suggest that you look at the Plot3d output in Smokeview and identify where in the domain the velocity went off the rails. That could provide a hint as to what is going wrong.

[FireResearch-BK]

Well, I kind of have an idea where in the domain the error occurred from the velocity and pressure errors, but this information is of little use since, in order to preserve restartability, the geometry can not be altered to harden the domain against excessive values.

I know I shouldn't restart after a crash that was not caused by power loss or inadvertent termination of FDS' processes, but I'm curious if a case can be recovered by other means than starting completely anew.

[mcgratta]

Even if you manage to restart the case, you may hit an instability again if the underlying problem is not addressed. When I debug cases that are submitted for analysis, I can sometimes change a few parameters or tighten tolerances that gets me past the point of the original instability, but just as a I write in the post that the problem is solved, another instability occurs. Sometimes a case runs successfully, but I consider it "fragile", meaning that a slight change in numerical or physical parameters sends the case just beyond the edge of stability. We see that when we run validation cases and 15 of 16 seemingly identical cases end successfully and one fails. This probably means that this case is fragile, and much of our development efforts are to make these cases less fragile.

[FireResearch-BK]

Yes, I'm aware of that. Starting anew with a more robust geometry is of course the best practice, but as I've said to Jo, the case fortunately only crashed past the point where it provides usable results. All I want to do here is see if there is anything happening that might be worth of mention in the report.

Tightening the tolerances for the FFT solver is a step I've skipped this time, as in a past case that became unstable, it did not work out (only using ULMAT did).

[FireResearch-BK]

I've played with the case some more and - for now at least - managed to recover it.

What I did:
Since the case only ran for 4 seconds from the restart point with UGLMAT HYPRE as a pressure solver before crashing (see #14940 (comment); this was later than the crash with FFT), I've simply set DT_RESTART=1.0 to create a restart point every second and restarted. This yielded a restart point before the UGLMAT HYPRE crash, but past the point at which FFT would crash. I then simply restarted the case from that point with the FFT solver and it's still running.

I'm not sure if UGLMAT HYPRE managed to clean up and tighten the velocities in the domain enough for FFT to continue, but so far, I'm not seeing anything out of the ordinary in any output file.

[uihiuy]

I am doing the solid fire spreading. The same grid size of 10mmpmma samples is still unstable after running again, but there is no numerical instability in 2mm and 5mm pmma samples. Why?

[FireResearch-BK]

I am doing the solid fire spreading. The same grid size of 10mmpmma samples is still unstable after running again, but there is no numerical instability in 2mm and 5mm pmma samples. Why?

This has absolutely nothing to do with this discussion. Please open a separate topic.

[FireResearch-BK]

This case is still running without any warnings whatsoever!

I've stopped it at some other point, but hat zero issues until then.

[mcgratta]

What did you change in the input file to make the restart successful?

[FireResearch-BK]

DT_RESTART=1.0 in &MISC line, deactivated the &PRES line with the configuration for FFT and added a &PRES SOLVER='UGLMAT HYPRE/'. Ran this until the case crashed after 2 seconds, but obtained a restart point.
After the crash with UGLMAT HYPRE, I simply deactivated the line for this solver, reactivated the line for the FFT solver configuration and set DT_RESTART=10.0.

[FireResearch-BK]

Just to demonstrate how randomly successful restarts are: I had another case, using the exact same geometry, design fire (just at another location), meshes, etc. as the problematic one from the first post running on my personal machine.
There were zero warnings and issues in the outputs. Due to some necessary computer maintenance, I had to stop it at ~1.200s where there is already a ton of smoke in the domain. The case was stopped nominally without any warnings and issues whatsoever.
I've transfered the case files to a (faster) workstation, but am completely unable to restart it. I've tried the UGLMAT HYPRE trick from #14940 (comment), I've tried ULMAT, I've tried extremely loose limits for min and max density, but to no avail.
The case always crashes due to numerical instability after 100 time steps without prior warning, even when I tried to restart it on my computer. This is absolutely weird.

In such a case, it would help if restart points generated by DT_RESTART=... were different to those generated by stopping the case. This will naturally increases data usage on the disk, but would at least provide a fallback restart point in case the one generated by stopping prevents the case from being restarted.

[mcgratta]

Restarts have always been fragile because we have to select all the variables to write out and read in explicitly. It is not as if we are just dumping whatever is in RAM onto disk, and then moving it back. In any event, there is no functional difference between the scheduled restarts and the user-specified restarts. In both instances, FDS continues marching in time until it gets to the time step where the radiation solver has completed so that the restarted case picks up where the radiation solver left off. Are you moving the restart files from one Windows computer to another, or Mac to Mac, Windows to Windows, or linux to linux? I would be concerned if the OS changed, even though I cannot tell you exactly why. I would worry about some subtle difference in the way the binary files are treated. I'm not sure that the Fortran standard demands that a binary file written out by DOS is expected to be read in by linux.

[FireResearch-BK]

It's Windows 11 24H2 to Windows 11 24H2, using the exact same FDS release (6.10.1).

It might not have been too clear from the previous post, but the restarted case for some reason also crashes on my machine, on which it ran originally. :/

[mcgratta]

So it may not be a problem of transferring files, but just a failure to read to the restart files. We have seen that DOS is more fragile than linux when it comes to restarts. There seems to be some line or file limit in DOS that makes starting large cases a problem. We've never pinpointed exactly what that is. My advice to Windows users is to not stop the jobs if at all possible, unless the size of the job is fairly small.

[FireResearch-BK]

I've just tried to restart the case on my machine to check the console outputs. It immediately throws "minimum density" warnings. Looking at the .out file (see excerpt in attachment), it seems that the pressure error before the stop was already pretty high, but multiplies by a factor of 1000 after the restart.

Console output is:

WARNING: Minimum density, 0.089 kg/m3, clipped in Mesh 7
WARNING: Minimum density, 0.089 kg/m3, clipped in Mesh 7
WARNING: Minimum density, 0.089 kg/m3, clipped in Mesh 7
WARNING: Minimum density, 0.089 kg/m3, clipped in Mesh 7
Time Step: 61686, Simulation Time: 1152.6154 s

ERROR(374): Numerical Instability - FDS stopped (CHID: [...])

Pressure solver configuration in the input file:

&PRES VELOCITY_TOLERANCE=0.005, MAX_PRESSURE_ITERATIONS=50, SUSPEND_PRESSURE_ITERATIONS=T/

outfile.txt

[FireResearch-BK]

Ah, I think I know what was going wrong. The FFT pressure solver suspended its iterations way too early with SUSPEND_PRESSURE_ITERATIONS=T/ (after 5 or so; default is 10) which was not nearly enough to keep the velocity and pressure errors in check.
I've removed this parameter to only leave the tolerance and maximum iterations, i.e.:

&PRES VELOCITY_TOLERANCE=0.005, MAX_PRESSURE_ITERATIONS=50/

Instead of a dozen "minimum pressure" warnings after the restart, I now only get two and after the first 200 time steps after the restart, the maximum velocity and pressure errors have decreased significantly. The pressure solver still hits max iterations (for now) and the CPU time per time step massively increased, but so far, things look much better than they did before.

I could wait until the pressure solver has driven down the errors enough, stop the case, reintroduce "SUSPEND_PRESSURE_ITERATIONS=T" and restart it, but I think I'll go for safety and leave the case running as it is.

- Edit:
Thinking about it, simply removing "SUSPEND_PRESSURE_ITERATIONS" and maybe increasing "MAX_PRESSURE_ITERATIONS" to 100 or so for a while might have been a more straightforward solution for that other case than switching to UGLMAT HYPRE (see #14940 (comment)).