Issue with `TimeStepWizard` on `GPU()`

[mncrowe]

Adding a timestep wizard when using a GPU appears to lead to a CUDA Kernel issue where something is requesting more resources than available. Tested on Oceananigans v0.95.11 with Julia 1.11.1. This bug persists on a different system with the same installations so it appears not to be an installation issue.

MWE:

using Oceananigans

grid = RectilinearGrid(GPU(), size=(100, 100), extent=(10, 10), topology=(Periodic, Periodic, Flat))
model = NonhydrostaticModel(; grid)
simulation = Simulation(model, Δt=1, stop_time=10)
conjure_time_step_wizard!(simulation)
run!(simulation)

Seems to be related to this.

[ Info: Initializing simulation...
ERROR: LoadError: Number of threads per block exceeds kernel limit (1024 > 896).
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] diagnose_launch_failure(f::CUDA.CuFunction, err::CUDA.CuError; blockdim::CUDA.CuDim3, threaddim::CUDA.CuDim3, shmem::Int64)
    @ CUDA ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:120
  [3] launch(::CUDA.CuFunction, ::CUDA.KernelState, ::CartesianIndices{…}, ::CartesianIndices{…}, ::CUDA.CuDeviceArray{…}, ::KernelFunctionOperation{…}; blocks::Int64, threads::Int64, cooperative::Bool, shmem::Int64, stream::CUDA.CuStream)
    @ CUDA ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:73
  [4] launch
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:52 [inlined]
  [5] #972
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:189 [inlined]
  [6] macro expansion
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:149 [inlined]
  [7] macro expansion
    @ ./none:0 [inlined]
  [8] convert_arguments
    @ ./none:0 [inlined]
  [9] #cudacall#971
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:191 [inlined]
 [10] cudacall
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:187 [inlined]
 [11] macro expansion
    @ ~/.julia/packages/CUDA/1kIOw/src/compiler/execution.jl:279 [inlined]
 [12] macro expansion
    @ ./none:0 [inlined]
 [13] (::CUDA.HostKernel{…})(::typeof(identity), ::typeof(min), ::Nothing, ::CartesianIndices{…}, ::CartesianIndices{…}, ::Val{…}, ::CUDA.CuDeviceArray{…}, ::KernelFunctionOperation{…}; convert::Val{…}, call_kwargs::@Kwargs{…})
    @ CUDA ./none:0
 [14] AbstractKernel
    @ ./none:0 [inlined]
 [15] macro expansion
    @ ~/.julia/packages/CUDA/1kIOw/src/compiler/execution.jl:114 [inlined]
 [16] mapreducedim!(f::typeof(identity), op::typeof(min), R::SubArray{…}, A::KernelFunctionOperation{…}; init::Nothing)
    @ CUDA ~/.julia/packages/CUDA/1kIOw/src/mapreduce.jl:271
 [17] mapreducedim!(f::typeof(identity), op::typeof(min), R::SubArray{…}, A::KernelFunctionOperation{…})
    @ CUDA ~/.julia/packages/CUDA/1kIOw/src/mapreduce.jl:169
 [18] mapreducedim!(f::Function, op::Function, R::SubArray{…}, A::KernelFunctionOperation{…})
    @ GPUArrays ~/.julia/packages/GPUArrays/uiVyU/src/host/mapreduce.jl:10
 [19] #minimum!#969
    @ ./reducedim.jl:1006 [inlined]
 [20] minimum!(f::Function, r::Field{…}, a::KernelFunctionOperation{…}; condition::Nothing, mask::Float64, kwargs::@Kwargs{…})
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/rp0vf/src/Fields/field.jl:677
 [21] minimum(f::Function, c::KernelFunctionOperation{…}; condition::Nothing, mask::Float64, dims::Function)
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/rp0vf/src/Fields/field.jl:707
 [22] minimum
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Fields/field.jl:696 [inlined]
 [23] minimum
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Fields/field.jl:716 [inlined]
 [24] cell_advection_timescale
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Advection/cell_advection_timescale.jl:16 [inlined]
 [25] cell_advection_timescale
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Models/NonhydrostaticModels/NonhydrostaticModels.jl:80 [inlined]
 [26] new_time_step
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/time_step_wizard.jl:103 [inlined]
 [27] TimeStepWizard
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/time_step_wizard.jl:117 [inlined]
 [28] (::Callback{…})(sim::Simulation{…})
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/callback.jl:15
 [29] initialize!(sim::Simulation{…})
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/run.jl:226
 [30] time_step!(sim::Simulation{…})
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/run.jl:118
 [31] run!(sim::Simulation{…}; pickup::Bool)
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/run.jl:102
 [32] run!(sim::Simulation{…})
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/run.jl:91
 [33] top-level scope
    @ /reserved/local/mcrowe/Julia_scripts/SI_transport/MWE.jl:7
 [34] include(fname::String)
    @ Main ./sysimg.jl:38
 [35] top-level scope
    @ REPL[2]:1
in expression starting at /reserved/local/mcrowe/Julia_scripts/SI_transport/MWE.jl:7

caused by: CUDA error: too many resources requested for launch (code 701, ERROR_LAUNCH_OUT_OF_RESOURCES)
Stacktrace:
  [1] throw_api_error(res::CUDA.cudaError_enum)
    @ CUDA ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/libcuda.jl:30
  [2] check
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/libcuda.jl:37 [inlined]
  [3] cuLaunchKernel
    @ ~/.julia/packages/CUDA/1kIOw/lib/utils/call.jl:34 [inlined]
  [4] (::CUDA.var"#966#967"{Bool, Int64, CUDA.CuStream, CUDA.CuFunction, CUDA.CuDim3, CUDA.CuDim3})(kernelParams::Vector{Ptr{…}})
    @ CUDA ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:66
  [5] macro expansion
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:33 [inlined]
  [6] macro expansion
    @ ./none:0 [inlined]
  [7] pack_arguments(::CUDA.var"#966#967"{…}, ::CUDA.KernelState, ::CartesianIndices{…}, ::CartesianIndices{…}, ::CUDA.CuDeviceArray{…}, ::KernelFunctionOperation{…})
    @ CUDA ./none:0
  [8] launch(::CUDA.CuFunction, ::CUDA.KernelState, ::CartesianIndices{…}, ::CartesianIndices{…}, ::CUDA.CuDeviceArray{…}, ::KernelFunctionOperation{…}; blocks::Int64, threads::Int64, cooperative::Bool, shmem::Int64, stream::CUDA.CuStream)
    @ CUDA ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:59
  [9] launch
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:52 [inlined]
 [10] #972
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:189 [inlined]
 [11] macro expansion
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:149 [inlined]
 [12] macro expansion
    @ ./none:0 [inlined]
 [13] convert_arguments
    @ ./none:0 [inlined]
 [14] #cudacall#971
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:191 [inlined]
 [15] cudacall
    @ ~/.julia/packages/CUDA/1kIOw/lib/cudadrv/execution.jl:187 [inlined]
 [16] macro expansion
    @ ~/.julia/packages/CUDA/1kIOw/src/compiler/execution.jl:279 [inlined]
 [17] macro expansion
    @ ./none:0 [inlined]
 [18] (::CUDA.HostKernel{…})(::typeof(identity), ::typeof(min), ::Nothing, ::CartesianIndices{…}, ::CartesianIndices{…}, ::Val{…}, ::CUDA.CuDeviceArray{…}, ::KernelFunctionOperation{…}; convert::Val{…}, call_kwargs::@Kwargs{…})
    @ CUDA ./none:0
 [19] AbstractKernel
    @ ./none:0 [inlined]
 [20] macro expansion
    @ ~/.julia/packages/CUDA/1kIOw/src/compiler/execution.jl:114 [inlined]
 [21] mapreducedim!(f::typeof(identity), op::typeof(min), R::SubArray{…}, A::KernelFunctionOperation{…}; init::Nothing)
    @ CUDA ~/.julia/packages/CUDA/1kIOw/src/mapreduce.jl:271
 [22] mapreducedim!(f::typeof(identity), op::typeof(min), R::SubArray{…}, A::KernelFunctionOperation{…})
    @ CUDA ~/.julia/packages/CUDA/1kIOw/src/mapreduce.jl:169
 [23] mapreducedim!(f::Function, op::Function, R::SubArray{…}, A::KernelFunctionOperation{…})
    @ GPUArrays ~/.julia/packages/GPUArrays/uiVyU/src/host/mapreduce.jl:10
 [24] #minimum!#969
    @ ./reducedim.jl:1006 [inlined]
 [25] minimum!(f::Function, r::Field{…}, a::KernelFunctionOperation{…}; condition::Nothing, mask::Float64, kwargs::@Kwargs{…})
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/rp0vf/src/Fields/field.jl:677
 [26] minimum(f::Function, c::KernelFunctionOperation{…}; condition::Nothing, mask::Float64, dims::Function)
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/rp0vf/src/Fields/field.jl:707
 [27] minimum
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Fields/field.jl:696 [inlined]
 [28] minimum
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Fields/field.jl:716 [inlined]
 [29] cell_advection_timescale
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Advection/cell_advection_timescale.jl:16 [inlined]
 [30] cell_advection_timescale
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Models/NonhydrostaticModels/NonhydrostaticModels.jl:80 [inlined]
 [31] new_time_step
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/time_step_wizard.jl:103 [inlined]
 [32] TimeStepWizard
    @ ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/time_step_wizard.jl:117 [inlined]
 [33] (::Callback{…})(sim::Simulation{…})
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/callback.jl:15
 [34] initialize!(sim::Simulation{…})
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/run.jl:226
 [35] time_step!(sim::Simulation{…})
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/run.jl:118
 [36] run!(sim::Simulation{…}; pickup::Bool)
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/run.jl:102
 [37] run!(sim::Simulation{…})
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/rp0vf/src/Simulations/run.jl:91
 [38] top-level scope
    @ /reserved/local/mcrowe/Julia_scripts/SI_transport/MWE.jl:7
 [39] include(fname::String)
    @ Main ./sysimg.jl:38
 [40] top-level scope
    @ REPL[2]:1
Some type information was truncated. Use `show(err)` to see complete types.

[glwagner]

Oy. Are there any topologies / sizes that work? Seeing the error helps for sure.

[mncrowe]

Oy. Are there any topologies / sizes that work? Seeing the error helps for sure.

Yup, total gridpoints (Nx x Ny) of 1024 or fewer works. Topology doesn't seem to matter. Errors added above.

[glwagner]

I think what we need to do is figure out how to reproduce this minimally without depending on Oceananigans, and then open an issue on CUDA or GPUArrays, probably

[glwagner]

What GPU was this error received on?

[glwagner]

This cannot be reproduced on a Titan V.

[mncrowe]

What GPU was this error received on?

> nvidia-smi
Thu Mar  6 22:56:01 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.08             Driver Version: 550.127.08     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A6000               Off |   00000000:3B:00.0 Off |                  Off |
| 30%   59C    P0             81W /  300W |       1MiB /  49140MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

[ali-ramadhan]

I believe this issue is the same as #4047 and yeah

Number of threads per block exceeds kernel limit (1024 > 896)

seems very GPU-dependent.

@mncrowe I'd be curious if this issue persists with the latest version of Oceananigans.jl and the master branch of CUDA.jl (which you can add/install with ] add CUDA#master). Also curious if you can share your environment: the outputs of versioninfo(), CUDA.versioninfo(), and ] status -m.

[mncrowe]

Default environment:

(@v1.11) pkg> st
Status `~/.julia/environments/v1.11/Project.toml`
  [e30172f5] Documenter v1.8.1
  [44ee3b1c] GeophysicalFlows v0.16.3
  [98e50ef6] JuliaFormatter v1.0.62
  [14b8a8f1] PkgTemplates v0.7.53
  [91a5bcdd] Plots v1.40.9

Project environment:

(@v1.11) pkg> activate .
  Activating project at `/reserved/local/mcrowe/Julia_scripts/SI_transport`

(SI) pkg> st
Status `/reserved/local/mcrowe/Julia_scripts/SI/Project.toml`
  [30363a11] NetCDF v0.12.1
  [9e8cae18] Oceananigans v0.95.19
  [d0ccf422] Oceanostics v0.15.1

Full manifest:

(SI) pkg> status -m
Status `/reserved/local/mcrowe/Julia_scripts/SI/Manifest.toml`
  [621f4979] AbstractFFTs v1.5.0
  [79e6a3ab] Adapt v4.2.0
  [a9b6321e] Atomix v1.1.1
  [ab4f0b2a] BFloat16s v0.5.0
  [fa961155] CEnum v0.5.0
  [179af706] CFTime v0.1.4
  [052768ef] CUDA v5.6.1
  [1af6417a] CUDA_Runtime_Discovery v0.3.5
  [3da002f7] ColorTypes v0.12.0
  [5ae59095] Colors v0.13.0
  [1fbeeb36] CommonDataModel v0.3.7
  [34da2185] Compat v4.16.0
  [a216cea6] CompoundPeriods v0.5.4
  [187b0558] ConstructionBase v1.5.8
  [a8cc5b0e] Crayons v4.1.1
  [7445602f] CubedSphere v0.3.1
  [9a962f9c] DataAPI v1.16.0
  [a93c6f00] DataFrames v1.7.0
  [864edb3b] DataStructures v0.18.20
  [e2d170a0] DataValueInterfaces v1.0.0
  [3c3547ce] DiskArrays v0.4.11
  [b4f34e82] Distances v0.10.12
  [ffbed154] DocStringExtensions v0.9.3
  [e2ba6199] ExprTools v0.1.10
  [7a1cc6ca] FFTW v1.8.1
  [5789e2e9] FileIO v1.17.0
  [53c48c17] FixedPointNumbers v0.8.5
  [0c68f7d7] GPUArrays v11.2.2
  [46192b85] GPUArraysCore v0.2.0
  [61eb1bfa] GPUCompiler v1.2.0
  [c27321d9] Glob v1.3.1
  [076d061b] HashArrayMappedTries v0.2.0
  [40713840] IncompleteLU v0.2.1
  [842dd82b] InlineStrings v1.4.3
  [41ab1584] InvertedIndices v1.3.1
  [42fd0dbc] IterativeSolvers v0.9.4
  [82899510] IteratorInterfaceExtensions v1.0.0
  [033835bb] JLD2 v0.5.11
  [692b3bcd] JLLWrappers v1.7.0
  [63c18a36] KernelAbstractions v0.9.34
  [929cbde3] LLVM v9.2.0
  [8b046642] LLVMLoopInfo v1.0.0
  [8ac3fa9e] LRUCache v1.6.1
  [b964fa9f] LaTeXStrings v1.4.0
  [da04e1cc] MPI v0.20.22
  [3da0fdf6] MPIPreferences v0.1.11
  [1914dd2f] MacroTools v0.5.15
  [e1d29d7a] Missings v1.2.0
  [78c3b35d] Mocking v0.8.1
  [85f8d34a] NCDatasets v0.14.6
⌅ [5da4648a] NVTX v0.3.5
  [30363a11] NetCDF v0.12.1
  [9e8cae18] Oceananigans v0.95.19
  [d0ccf422] Oceanostics v0.15.1
  [6fe1bfb0] OffsetArrays v1.15.0
  [bac558e1] OrderedCollections v1.8.0
  [eebad327] PkgVersion v0.3.3
  [2dfb63ee] PooledArrays v1.4.3
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [08abe8d2] PrettyTables v2.4.0
  [94ee1d12] Quaternions v0.7.6
  [74087812] Random123 v1.7.0
  [e6cf234a] RandomNumbers v1.6.0
  [c1ae055f] RealDot v0.1.0
  [3cdcf5f2] RecipesBase v1.3.4
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.1
  [6038ab10] Rotations v1.7.1
  [7e506255] ScopedValues v1.3.0
  [6c6a2e73] Scratch v1.2.1
  [d496a93d] SeawaterPolynomials v0.3.5
  [91c51154] SentinelArrays v1.4.8
  [a2af1166] SortingAlgorithms v1.2.1
  [90137ffa] StaticArrays v1.9.13
  [1e83bf80] StaticArraysCore v1.4.3
  [10745b16] Statistics v1.11.1
  [82ae8749] StatsAPI v1.7.0
  [892a3eda] StringManipulation v0.4.1
  [09ab397b] StructArrays v0.7.0
  [dc5dba14] TZJData v1.4.0+2025a
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.12.0
  [6aa5eb33] TaylorSeries v0.18.3
  [f269a46b] TimeZones v1.21.3
  [a759f4b9] TimerOutputs v0.5.28
  [bdfc003b] TimesDates v0.3.1
  [3bb67fe8] TranscodingStreams v0.11.3
  [013be700] UnsafeAtomics v0.3.0
  [0b7ba130] Blosc_jll v1.21.6+2
  [6e34b625] Bzip2_jll v1.0.9+0
⌅ [4ee394cb] CUDA_Driver_jll v0.10.4+0
⌅ [76a88914] CUDA_Runtime_jll v0.15.5+0
  [f5851436] FFTW_jll v3.3.10+3
⌅ [0234f1f7] HDF5_jll v1.14.2+1
  [e33a78d0] Hwloc_jll v2.12.0+0
  [1d5cc7b8] IntelOpenMP_jll v2025.0.4+0
  [9c1d0b0a] JuliaNVTXCallbacks_jll v0.2.1+0
  [dad2f222] LLVMExtra_jll v0.0.35+0
  [1d63c593] LLVMOpenMP_jll v18.1.7+0
  [94ce4f54] Libiconv_jll v1.18.0+0
  [5ced341a] Lz4_jll v1.10.1+0
  [856f044c] MKL_jll v2025.0.1+1
  [7cb0a576] MPICH_jll v4.3.0+1
  [f1f71cc9] MPItrampoline_jll v5.5.3+0
  [9237b28f] MicrosoftMPI_jll v10.1.4+3
  [e98f9f5b] NVTX_jll v3.1.1+0
⌃ [7243133f] NetCDF_jll v400.902.209+0
  [fe0851c0] OpenMPI_jll v5.0.7+0
  [458c3c95] OpenSSL_jll v3.0.16+0
  [02c8fc9c] XML2_jll v2.13.6+1
  [ffd25f8a] XZ_jll v5.6.4+1
  [3161d3a3] Zstd_jll v1.5.7+1
  [1e29f10c] demumble_jll v1.3.0+0
  [477f73a3] libaec_jll v1.1.3+0
  [337d8026] libzip_jll v1.11.3+0
  [1317d2d5] oneTBB_jll v2022.0.0+0
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [8ba89e20] Distributed v1.11.0
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching v1.11.0
  [9fa8497b] Future v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [4af54fe1] LazyArtifacts v1.11.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [37e2e46d] LinearAlgebra v1.11.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [a63ad114] Mmap v1.11.0
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.11.0
  [de0858da] Printf v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [6462fe0b] Sockets v1.11.0
  [2f01184e] SparseArrays v1.11.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.1.1+0
  [deac9b47] LibCURL_jll v8.6.0+0
  [e37daf67] LibGit2_jll v1.7.2+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.6+0
  [14a3606d] MozillaCACerts_jll v2023.12.12
  [4536629a] OpenBLAS_jll v0.3.27+1
  [bea87d4a] SuiteSparse_jll v7.7.0+0
  [83775a58] Zlib_jll v1.2.13+1
  [8e850b90] libblastrampoline_jll v5.11.0+0
  [8e850ede] nghttp2_jll v1.59.0+0
  [3f19e933] p7zip_jll v17.4.0+2
Info Packages marked with ⌃ and ⌅ have new versions available. Those with ⌃ may be upgradable, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated -m`

Versions:

julia> versioninfo()
Julia Version 1.11.3
Commit d63adeda50d (2025-01-21 19:42 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 64 × Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, cascadelake)
Threads: 1 default, 0 interactive, 1 GC (on 64 virtual cores)

Can't currently get CUDA.versioninfo() as NVML was very recently updated so system needs a restart (outside my control). I'll try installing CUDA from the master branch later today on an identical machine.

Update: all identical machines in need of the same restart. There doesn't seem any point for me trying to test things as they are without being able to rule out driver issues as the problem.

[ali-ramadhan]

Thanks! Just a heads up that I see you're using Julia 1.11 but 1.11 is not fully supported yet (see #3836) so you should use 1.10.8.

I don't think 1.11 is the cause of this current issue, but it could cause other issues.

[mncrowe]

Thanks! Just a heads up that I see you're using Julia 1.11 but 1.11 is not fully supported yet (see #3836) so you should use 1.10.8.

I don't think 1.11 is the cause of this current issue, but it could cause other issues.

Thanks, I was aware there were some issues but the scripts I had were running as intended on 1.11 so I didn't bother downgrading. Will use 1.10.8 for future testing of this issue.

[ali-ramadhan]

@mncrowe That's unfortunate about the driver issues. Just a heads up that @simone-silvestri had some good luck with CUDA#master in #4139 (comment) so it might fix this issue (and #4047 as well)!

[mncrowe]

@mncrowe That's unfortunate about the driver issues. Just a heads up that @simone-silvestri had some good luck with CUDA#master in #4139 (comment) so it might fix this issue (and #4047 as well)!

Thanks @ali-ramadhan, it seems that using CUDA#master does fix this issue, at least for the example I posted above. Will need to check things work out for my full simulation but it looks like this is the solution.

[simone-silvestri]

Note that the last release of CUDA should be sufficient (CUDA@5.7.0)