low_Mach for HLL Riemann solver and 6-equation model

[hyeoksu-lee]

Description

This PR extends low_Mach feature to HLL Riemann solver and 6-equation model. Previously, it only covered HLLC Riemann solver and 5-equation model. Newly added combinations are listed below

• HLL Riemann solver + 5-equation model (low_Mach == 1)
• HLLC Riemann solver + 6-equation model (low_Mach == 1 and 2)

Also, this PR includes a minor fix to omega_wrt(3) in post_process. One should be able to compute omega_wrt(3) even in 2D flow, which is not allowed in the current master branch. I found this bug as I validate low_Mach using omega_wrt(3) in 2D flow.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Scope

• This PR comprises a set of related changes with a common goal

If you cannot check the above box, please split your PR into multiple PRs that each have a common goal.

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.
Provide instructions so we can reproduce.
Please also list any relevant details for your test configuration

• HLL+5eq: 2D Gresho vortex
  For HLL + 5eq, low_Mach == 2 does not work.
  

• HLLC+6eq: 2D mixing layer
  

Test Configuration:

• What computers and compilers did you use to test this: Carpenter CPU for above cases; Delta GPU for GPU test

Checklist

• I have added comments for the new code
• I added Doxygen docstrings to the new code
• I have made corresponding changes to the documentation (docs/)
• I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
• I have added example cases in examples/ that demonstrate my new feature performing as expected.
  They run to completion and demonstrate "interesting physics"
• I ran ./mfc.sh format before committing my code
• New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
• This PR does not introduce any repeated code (it follows the DRY principle)
• I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

• Checked that the code compiles using NVHPC compilers
• Checked that the code compiles using CRAY compilers
• Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
• Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
• Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
• Ran an Omniperf profile using ./mfc.sh run XXXX --gpu -t simulation --omniperf, and have attached the output file and plain text results to this PR.
• Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature

[codecov]

Codecov Report

Attention: Patch coverage is 47.61905% with 11 lines in your changes missing coverage. Please review.

Project coverage is 43.15%. Comparing base (40164cf) to head (1985a4e).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
src/simulation/m_riemann_solvers.fpp	31.25%	8 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #833      +/-   ##
==========================================
- Coverage   43.15%   43.15%   -0.01%     
==========================================
  Files          68       68              
  Lines       20262    20264       +2     
  Branches     2424     2429       +5     
==========================================
- Hits         8745     8744       -1     
  Misses      10054    10054              
- Partials     1463     1466       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hyeoksu-lee]

Benchmark / Georgia Tech | Phoenix (NVHPC) (cpu) (pull_request) failed with the following error:

Error: Failed to load YAML from "../master/bench-cpu.yaml": [Errno 2] No such file or directory: '../master/bench-cpu.yaml'

It looks like not from the PR, but from the master branch. Do you have any idea? @sbryngelson

[wilfonba]

@hyeoksu-lee I restarted the jobs. The runners on that system have been failing spuriously lately and we haven't figured out why yet.

[hyeoksu-lee]

@wilfonba Great, thanks!

[sbryngelson]

@hyeoksu-lee is this PR complete/ready for review?

[hyeoksu-lee]

@sbryngelson Yes, it's ready for review. I haven't completed some GPU checklists though. The code change is minimal, so I think it barely affects GPU performance. If you want me to check them to make sure, please let me know!

[sbryngelson]

This looked good, but ChatGPT found a few things that should maybe be fixed/changed. I wouldn't implement the one that involves --fastmath intrinsics, though it may have a point about the computing pcorr once thing.

The divide-by-zero thing is interesting. I think it could be a good addition in the future because many NaNs start from this part of the code, broadly speaking. But I don't think this is the right PR for adding that -- it could introduce some breaking changes.

See this link: https://chatgpt.com/share/6829cb31-7890-800a-a903-449b6ba5ddbd

[hyeoksu-lee]

@sbryngelson Very interesting that ChatGPT can review the PR like this! However, I think ChatGPT misunderstood one thing here.

It stated:

[FIX] Additional (s_M/s_L)*(s_P/s_R) factors involve divisions by wave speeds that may be zero or nearly zero in true low-Mach limits, re-introducing singularities the original correction is meant to avoid. Guard with eps_wp:

However, in true low Mach limits, s_L and s_R are indeed close to speed of sound in magnitude, not zero, as they are defined as follows:

s_L = min(vel_L(dir_idx(1)) - c_L, vel_R(dir_idx(1)) - c_R)
s_R = max(vel_R(dir_idx(1)) + c_R, vel_L(dir_idx(1)) + c_L)

where vel_L/R << c_L/R in low Mach limit. Therefore, I believe this part will not introduce division-by-zero.

[sbryngelson]

Yeah I'm not worried about that one

[hyeoksu-lee]

Also, I think other suggestions by ChatGPT seem not really improve the code except for redundant n > 0 for omega_wrt, which is now removed. n == 0 is prohibited by checker anyway.