NetCDF output error on Kestrel GPUs

[sbidadi9]

Bug description

When using more than one GPU node, writing AMR-Wind boundary data in NETCDF format generates errors:

Below is the output after one iteration of a multiphase simulation:

This issue does not occur when running the same case on a single GPU node:

AMR-Wind information

==============================================================================
AMR-Wind (https://github.com/exawind/amr-wind)

AMR-Wind version :: v3.4.0-30-gfe3799a1
AMR-Wind Git SHA :: fe3799a
AMReX version :: 25.02-23-g06b4a5b105f5

Exec. time :: Thu Apr 3 22:56:34 2025
Build time :: Apr 2 2025 09:50:56
C++ compiler :: NVHPC 23.9.0

MPI :: ON (Num. ranks = 4)
GPU :: ON (Backend: CUDA)
OpenMP :: OFF

Enabled third-party libraries:
NetCDF 4.9.2
``

[jrood-nrel]

Can you try it with export CUDA_LAUNCH_BLOCKING=1 set?

[sbidadi9]

@jrood-nrel I will run the case again with the above setting.

[github-actions]

This issue is stale because it has been open 30 days with no activity.