How to Setup MultiRegion Simulation

[kenflat2]

I am running a simulation at 1km temporal resolution which amounts to a 2000x2000x76 size grid. This saturates a single GPU so I would like to run on multiple GPUs.

I modified my grid setup to

grid = RectilinearGrid( GPU(), size=(nx, ny, nz),
                        x=(0, Lx),
                        y=(0, Ly),
                        z= Φ,
                        topology=(Periodic, Bounded, Bounded),
                        halo=(7,7,7));

free_surface = SplitExplicitFreeSurface(grid; substeps=3)

multi_grid = MultiRegionGrid(grid, partition=XPartition(2), devices=(3, 4))

immersed_multi_grid = ImmersedBoundaryGrid(multi_grid, GridFittedBottom(bottom_height));

but when I pass the immersed_multi_grid into my HydrostaticFreeSurfaceModel I get the strange error ERROR: type MultiRegionObject has no field west. Is there something incorrect my setup here?

I also ask because the validation example systems that do an ImmersedBoundaryGrid with multiple regions may be out of date. The multi_region_turbulence.jl file first immerses the grid then splits it into regions

grid = ImmersedBoundaryGrid(grid, GridFittedBottom(bottom))
mrg  = MultiRegionGrid(grid, partition=XPartition(2), devices = 2)

but this ordering didn't work when I tried it.

Also, is it appropriate to create the SplitExplicitFreeSurface using the underlying_grid instead of the ultimate immersed_multi_grid?

Thank you for any help.

[glwagner]

To distributed your simulation among GPUs, use the Distributed(GPU()) architecture. There's more information here: https://clima.github.io/OceananigansDocumentation/stable/grids/