Implement scale-invariant dynamic Smagorinsky (#3642)

* got the bones in place

* bugfix and added first attempt at AveragingProcedure

* finally working

* added validation script

* bit of cleanup

* change abstract type names and add inbounds

* simpler averaging type

* bugfix in Base.summary

* cleanup

* Update src/TurbulenceClosures/turbulence_closure_implementations/scale_invariant_smagorinsky.jl

Co-authored-by: Gregory L. Wagner <[email protected]>

* change calculation of averaged fields

* automate validation script

* start with well-resolved noise

* plot Smagorinsky coefficient

* bugfix in validation

* rename validation file

* forgot inbounds macro

* fix interpolation

* simpler notation for names

* fewer calculations

* delete extra line

* shortcutuser shortcut for directional averaging

* add update_frequency to ScaleInvariantSmag

* added a couple of validation scripts

* put wall flow into a function and added some plotting code

* finished plotting in wall_flow

* removed python script

* added stop_time to options

* added type for averaging

* changed order of types

* abstracted update_frequency type

* run a test case only

* made it work on GPUs

* pass velocities to DiffusivityFields

* bugfix

* bugfix

* rename update_interval

* bugfix

* bugfix and cleanup

* better IC and difference scheme in wall_flow

* plot 2D snapshots and 1D means

* longer stop_time

* update validation scripts

* improve plotting 3d turbulence validation

* change advection to WENO

* Generalize SmagorinskyLilly to have averaged coefficients

* Use const AG = AbstractGrid

* Speed up by writing out LM MM more

* Add optional dims to DirectionalAveraging

* fix simple_smag.jl and add more @time statements

* adapt 3d_turbulence_smagorinsky.jl to new style

* cleanup old code

* first step towards optimization

* further remove strain calculation

* remove extra interpolation

* pre-calculate strain rates

* don't modify DiffusivityFields

* standardize notation for strain rate

* show methods for DirectionallyAvaragedCoefficient

* retest validation scripts

* fix validation

* generalize filter using interpolations and improve nomenclature

* Move Smagorinsky to dedicated module plus rename

* Remove SmagorinskyLilly

* Implement LagrangianAveraging

* Update src/TurbulenceClosures/turbulence_closure_implementations/Smagorinskys/scale_invariant_operators.jl

Co-authored-by: Simone Silvestri <[email protected]>

* Update src/TurbulenceClosures/turbulence_closure_implementations/Smagorinskys/scale_invariant_operators.jl

Co-authored-by: Simone Silvestri <[email protected]>

* Fix operator bugs

* Import with_Traers

* Import tracer diffusivities

* Fix LillyCoefficient constructor bug

* Fix another bug in LillyCoefficient

* another lilly coeff bug

* Add visualization script

* Add comparison script

* Special-case SmagLilly

* and show and summary methods for Smagorinsky closure and coefficients

* fix docs and time stepping tests

* added docstrings and exported new constructors

* clarifications for lagrangian averaging procedure

* add tests and improve validation

* Update viz script

* more tests for dynamic smagorinskys

* more tests for dynamic smagorinskys

* cleanup

* no need to export at the submodule level

* clean up old comments

* remove obsolete script

* Improve formatting

* Better syntax with DynamicSmagorinsky

* Get name into TurbulenceClosures

* Defaults

* fix test

* formatting and correct info message for test

* add info message

* more detailed info

* dig into whats causing the hopld up

* another info statement

* change info

* change up info statements

* more changing info statements

* shift info statements around a bit more

* restrict tests to catch error

* uncomment steps line

* no distributed tests

* more info statements

* more info statements

* more info statements

* shift info statements

* shift info again

* skip _compute_ll_MM! launch

* simplify _compute_LM_MM

* try another definition

* does a simplified version work?

* a bit more complexity

* another variation

* another variation

* another variation

* simplify call to off diagonal Mij

* cleanup and simplify formulation with constants

* avoid nested call

* revert changes

* add back GPU test

* forgot to initialize gpu env

* test time stepping

* remove info statements

* skip a couple of DynamicSmag tests

* import dynamic smag

* skip another dynamic smag test

* fix test

* add back into statement

* skip another test

* bring back tests

* bring back distributed tests

* add dependencies back

* Update test/test_time_stepping.jl

Co-authored-by: Gregory L. Wagner <[email protected]>

* Update test/test_turbulence_closures.jl

Co-authored-by: Gregory L. Wagner <[email protected]>

---------

Co-authored-by: Gregory L. Wagner <[email protected]>
Co-authored-by: Simone Silvestri <[email protected]>
Co-authored-by: glwagner <[email protected]>

Author: 3 people

.buildkite/pipeline.yml

@@ -23,7 +23,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
 
   - label: "🏕️ initialize cpu environment"
@@ -44,7 +44,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
 
 #####
@@ -63,7 +63,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -78,7 +78,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -98,7 +98,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -113,7 +113,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -129,7 +129,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -144,7 +144,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -160,7 +160,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -175,7 +175,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -191,7 +191,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -206,7 +206,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -226,7 +226,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -241,7 +241,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -261,7 +261,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -276,7 +276,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -296,7 +296,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -311,7 +311,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -331,7 +331,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -346,7 +346,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -366,7 +366,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -381,7 +381,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -401,7 +401,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -416,7 +416,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -436,7 +436,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -451,7 +451,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -471,7 +471,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -486,7 +486,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -506,7 +506,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -521,7 +521,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -541,7 +541,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -556,7 +556,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -576,7 +576,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -591,7 +591,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -607,7 +607,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -622,7 +622,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -642,7 +642,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -657,7 +657,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 
@@ -677,7 +677,7 @@ steps:
       architecture: GPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_gpu"
 
@@ -692,7 +692,7 @@ steps:
       architecture: CPU
     retry:
       automatic:
-        - exit_status: 1      
+        - exit_status: 1
           limit: 1
     depends_on: "init_cpu"
 

docs/oceananigans.bib

@@ -967,4 +967,15 @@ @article{Shchepetkin2005
   pages={347--404},
   year={2005},
   doi={10.1016/j.ocemod.2004.08.002}
-}
+}
+
+@article{BouZeid05,
+    author = {Bou-Zeid, Elie and Meneveau, Charles and Parlange, Marc},
+    title = "{A scale-dependent Lagrangian dynamic model for large eddy simulation of complex turbulent flows}",
+    journal = {Physics of Fluids},
+    volume = {17},
+    number = {2},
+    pages = {025105},
+    year = {2005},
+    doi = {10.1063/1.1839152},
+}

docs/src/model_setup/turbulent_diffusivity_closures_and_les_models.md

@@ -65,7 +65,9 @@ To use the default Smagorinsky-Lilly LES closure, we write
 julia> using Oceananigans.TurbulenceClosures
 
 julia> closure = SmagorinskyLilly()
-SmagorinskyLilly: C=0.16, Cb=1.0, Pr=1.0
+Smagorinsky closure with
+├── coefficient = LillyCoefficient(smagorinsky = 0.16, reduction_factor = 1.0)
+└── Pr = 1.0
 ```
 
 The parameters `C`, `Cb`, and `Pr` may alternatively be specified explicitly.