Specify CUDA version for login node builds

[max-Hawkins]

Description

Currently, specifying the MFC_CUDA_CC flag can cause builds to fail on login nodes where the system can't determine the CUDA version to build for. This adds a compiler flag to specify the CUDA version explicitly. To ensure that rearranging the build process doesn't break, I force CMAKE to find CUDA when the MFC_CUDA_CC flag is set.

Fixes #(687)

Fixes #687

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Something else

Scope

• This PR comprises a set of related changes with a common goal

If you cannot check the above box, please split your PR into multiple PRs that each have a common goal.

How Has This Been Tested?

Load modules for specific system
rm -rf build/staging
export MFC_CUDA_CC=XX where XX is a compute capability
Build GPU MFC on Phoenix and Delta

• What computers and compilers did you use to test this:
  See above

Checklist

• I have added comments for the new code
• I added Doxygen docstrings to the new code
• I have made corresponding changes to the documentation (docs/)
• I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
• I have added example cases in examples/ that demonstrate my new feature performing as expected.
  They run to completion and demonstrate "interesting physics"
• I ran ./mfc.sh format before committing my code
• New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
• This PR does not introduce any repeated code (it follows the DRY principle)
• I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

• Checked that the code compiles using NVHPC compilers
• Checked that the code compiles using CRAY compilers
• Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
• Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
• Enclosed the new feature via nvtx ranges so that they can be identified in profiles
• Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
• Ran an Omniperf profile using ./mfc.sh run XXXX --gpu -t simulation --omniperf, and have attached the output file and plain text results to this PR.
• Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature

[max-Hawkins]

Mat Colgrove's advice: https://forums.developer.nvidia.com/t/cannot-compile-on-login-node/282759

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 42.99%. Comparing base (6fe815d) to head (8c717df).
Report is 4 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #697   +/-   ##
=======================================
  Coverage   42.99%   42.99%           
=======================================
  Files          61       61           
  Lines       16053    16053           
  Branches     1796     1796           
=======================================
  Hits         6902     6902           
  Misses       8167     8167           
  Partials      984      984           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sbryngelson]

@max-Hawkins let me know the status of this. Will merge when it is ready (it passed Phoenix tests), just need to confirm it works fine on Delta - perhaps you already checked this.

[max-Hawkins]

Tested on both Delta and Phoenix after removing the entire build dir. All we had to do was set the NVHPC_CUDA_HOME environment variable. It just took me longer than it should have to realize this.