Low mach number correction for HLLC Riemann solver

[hyeoksu-lee]

Description

The HLLC Riemann solver is known to have an accuracy issue at the low Mach number limit. The new feature addresses this issue using two different methods (Chen et al. 2022 and Thornber et al. 2008)

Type of change

• New feature (non-breaking change which adds functionality)

Scope

• This PR comprises a set of related changes with a common goal

If you cannot check the above box, please split your PR into multiple PRs that each have a common goal.

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.
Provide instructions so we can reproduce.
Please also list any relevant details for your test configuration

• 2D_GreshoVortex for Ma = 0.001 (added to examples)

Test Configuration: 2D Gresho vortex is an exact solution to the incompressible N-S equations. The flow field should remain the same unless there is a numerical dissipation. The figure attached below shows the simulation results at Mach number 0.001 with low_Mach = 0 (no correction), low_Mach = 1 (Chen et al. 2022) and low_Mach = 2 (Thornber et al. 2008). Both correction methods work well.

• What computers and compilers did you use to test this:

1. Richardson (cpu)
2. Bridges2 (cpu)
3. Delta (cpu/A100x4)

Checklist

• I have added comments for the new code
• I added Doxygen docstrings to the new code
• I have made corresponding changes to the documentation (docs/)
• I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
• I have added example cases in examples/ that demonstrate my new feature performing as expected.
  They run to completion and demonstrate "interesting physics"
• I ran ./mfc.sh format before committing my code
• New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
• This PR does not introduce any repeated code (it follows the DRY principle)
• I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

• Checked that the code compiles using NVHPC compilers
• Checked that the code compiles using CRAY compilers
• Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
• Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)

I don't have access to MI200+ GPUs.

• Enclosed the new feature via nvtx ranges so that they can be identified in profiles
• Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
  nsys.zip
• Ran an Omniperf profile using ./mfc.sh run XXXX --gpu -t simulation --omniperf, and have attached the output file and plain text results to this PR.

I don't have access to machines that work with omniperf profile, I guess.

• Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature

For this one, should I turn on low_Mach feature on examples/3D_performance_test and compare with low_Mach = 0 (off) results for 1/2/8 GPUs?

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.

Project coverage is 58.82%. Comparing base (24ea4af) to head (506d671).
Report is 2 commits behind head on master.

Files	Patch %	Lines
src/simulation/m_checker.fpp	50.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #538      +/-   ##
==========================================
+ Coverage   58.80%   58.82%   +0.02%     
==========================================
  Files          57       57              
  Lines       14555    14570      +15     
  Branches     1913     1917       +4     
==========================================
+ Hits         8559     8571      +12     
- Misses       5437     5440       +3     
  Partials      559      559              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hyeoksu-lee]

@sbryngelson Could you re-run the benchmark test? It failed with exit code 1 without any useful messages. If it is really a problem, do you have any idea what went wrong? Thanks!