refactor(createpackages): use jinja for mf6 module code generation (#2333)

Reimplement createpackages.py with jinja. The aim is to match the old code generation capability without relying on mfstructure.py. This is a first step toward a leaner python representation of the mf6 input specification.

Note: mfstructure.py is still used at runtime which will need to be unraveled in followup work.

The new code generation machinery lives in flopy.mf6.utils.codegen. It includes a bunch of filters handling quirks of the generated classes which we can aim to eliminate in future. The templates should also get much simpler in future.

The module now consumes TOML rather than legacy DFNs. modflow-devtools is used at code generation time to convert DFNs to TOML, then generate sources as before. The conversion happens behind the scenes and does not change anything for the user of the codegen utility.

A new optional dependency group 'codegen' is introduced, with Jinja2 and modflow-devtools, which are required for the code generation utilities. These and some transitive dependencies (tomli/tomli_w) are added to environment.yml.

Miscellaneous:

- expand and add some mermaid diagrams to the MF6 module dev guide
- minor fix in flopy/mf6/data/mfdatastorage.py to avoid referencing a variable before it exists
- update DFNs as per MODFLOW-ORG/modflow6#2031

Author: wpbonelli

.docs/md/generate_classes.md

@@ -10,7 +10,9 @@
 
 MODFLOW 6 input continues to evolve as new models, packages, and options are developed, updated, and supported. All MODFLOW 6 input is described by DFN (definition) files, which are simple text files that describe the blocks and keywords in each input file. These definition files are used to build the input and output guide for MODFLOW 6. These definition files are also used to automatically generate FloPy classes for creating, reading and writing MODFLOW 6 models, packages, and options.  FloPy and MODFLOW 6 are kept in sync by these DFN (definition) files, and therefore, it may be necessary for a user to update FloPy using a custom set of definition files, or a set of definition files from a previous release.
 
-The FloPy classes for MODFLOW 6 are largely generated by a utility which converts DFN files in a modflow6 repository on GitHub or on the local machine into Python source files in your local FloPy install. For instance (output much abbreviated):
+The FloPy classes for MODFLOW 6 are largely generated by a utility which converts DFN files in a modflow6 repository on GitHub or on the local machine into Python source files in your local FloPy install.
+
+**Note**: to use this functionality, the `codegen` optional dependency group must be installed.
 
 ```bash
 $ python -m flopy.mf6.utils.generate_classes

.github/workflows/examples.yml

@@ -34,7 +34,9 @@ jobs:
             powershell
 
       - name: Install FloPy
-        run: pip install .
+        run: |
+          pip install .
+          pip install ".[codegen]"
 
       - name: OpenGL workaround on Linux
         if: runner.os == 'Linux'

.github/workflows/rtd.yml

@@ -17,7 +17,7 @@ concurrency:
   cancel-in-progress: true
 jobs:
   set_options:
-    name: Set release options
+    name: Set options
     runs-on: ubuntu-22.04
     outputs:
       ref: ${{ steps.set_ref.outputs.ref }}

.gitignore

@@ -105,5 +105,8 @@ app
 # DFN backups
 flopy/mf6/data/dfn_backup/
 
+# DFN TOML dir
+flopy/mf6/data/toml/
+
 # uv lockfile
-uv.lock
+uv.lock

docs/mf6_dev_guide.md

@@ -1,47 +1,79 @@
-Introduction
------------------------------------------------
+# Developing FloPy for MF6
 
-This file provides an overview of how FloPy for MODFLOW 6 (FPMF6) works under the hood and is intended for anyone who wants to add a new package, new model type, or new features to this library.  FloPy library files that support MODFLOW 6 can be found in the flopy/mf6 folder and sub-folders. 
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
 
-Package Meta-Data and Package Files
------------------------------------------------
+- [Introduction](#introduction)
+- [Code generation](#code-generation)
+- [Input specification](#input-specification)
 
-FPMF6 uses meta-data files located in flopy/mf6/data/dfn to define the model and package types supported by MODFLOW 6.  When additional model and package types are added to MODFLOW 6, additional meta-data files can be added to this folder and flopy/mf6/utils/createpackages.py can be run to add new packages to the FloPy library.  createpackages.py uses flopy/mf6/data/mfstructure.py to read meta-data files (*.dfn) and use that meta-data to create the package files found in flopy/mf6/modflow (do not directly modify any of the files in this folder, they are all automatically generated).  The automatically generated package files contain an interface for accessing package data and data documentation generated from the meta-data files.  Additionally, meta-data describing package data types and shapes is stored in the dfn attribute.  flopy/mf6/data/mfstructure.py can load structure information using the dfn attribute (instead of loading it from the meta-data files).  This allows for flopy to be installed without the dfn files.
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
-All meta-data can be accessed from the flopy.mf6.data.mfstructure.MFStructure class.  This is a singleton class, meaning only one instance of this class can be created.  The class contains a sim_struct attribute (which is a flopy.mf6.data.mfstructure.MFSimulationStructure object) which contains all of the meta-data for all package files.  Meta-data is stored in a structured format. MFSimulationStructure contains MFModelStructure and MFInputFileStructure objects, which contain the meta-data for each model type and each "simulation-level" package (tdis, ims, ...).  MFModelStructure contains model specific meta-data and a MFInputFileStructure object for each package in that model.  MFInputFileStructure contains package specific meta-data and a MFBlockStructure object for each block contained in the package file.  MFBlockStructure contains block specific meta-data and a MFDataStructure object for each data structure defined in the block, and MFDataStructure contains data structure specific meta-data and a MFDataItemStructure object for each data item contained in the data structure.  Data structures define the structure of data that is naturally grouped together, for example, the data in a numpy recarray.  Data item structures define the structure of specific pieces of data, for example, a single column of a numpy recarray.  The meta-data defined in these classes provides all the information FloPy needs to read and write MODFLOW 6 package and name files, create the Flopy interface, and check the data for various constraints.
+## Introduction
 
+This file provides an overview of how FloPy's MODFLOW 6 module `flopy.mf6` works under the hood. It is intended for FloPy maintainers, as well as anyone who wants to add a new package, new model, or new features to this library.
 
-***
-MFStructure --+ MFSimulationStructure --+ MFModelStructure --+ MFInputFileStructure --+ MFBlockStructure --+ MFDataStructure --+ MFDataItemStructure
+## Code generation
 
-Figure 1: FPMF6 generic data structure classes.  Lines connecting classes show a relationship defined between the two connected classes.  A "*" next to the class means that the  class is a sub-class of the connected class.  A "+" next to the class means that the class is contained within the connected class.
-***
+MODFLOW 6 describes its input specification with definition (DFN) files.
 
-Package and Data Base Classes
------------------------------------------------
+Definition files describe components (e.g. simulations, models, packages) in the MODFLOW 6 input hierarchy. Definition files are used to generate both source code and documentation.
 
-The package and data classes are related as shown below in figure 2.  On the top of the figure 2 is the MFPackage class, which is the base class for all packages.  MFPackage contains generic methods for building data objects and reading and writing the package to a file.  MFPackage contains a MFInputFileStructure object that defines how the data is structured in the package file.  MFPackage also contains a dictionary of blocks (MFBlock).  The MFBlock class is a generic class used to represent a block within a package.  MFBlock contains a MFBlockStructure object that defines how the data in the block is structured.  MFBlock also contains a dictionary of data objects (subclasses of MFData) contained in the block and a list of block headers (MFBlockHeader) for that block.  Block headers contain the block's name and optionally data items (eg. iprn).
+FloPy can generate a MODFLOW 6 compatibility layer for itself, given a set of definition files:
+
+- `flopy/mf6/utils/createpackages.py`: assumes definition files are in `flopy/mf6/data/dfn`
+- `flopy/mf6/utils/generate_classes.py`: downloads DFNs then runs `createpackages.py`
+
+For instance, to sync with DFNs from the MODFLOW 6 develop branch:
 
+```shell
+python -m flopy.mf6.utils.generate_classes --ref develop --no-backup
+```
 
-***
-MFPackage --+ MFBlock --+ MFData
+Generated files are created in `flopy/mf6/modflow/`.
 
-MFPackage --+ MFInputFileStructure
+The code generation utility downloads DFN files, loads them, and uses Jinja to generate corresponding source files. A definition file typically maps 1-1 to a source file and component class, but 1-many is also possible (e.g. a model definition file yields a model class/file and namefile package class/file).
 
-MFBlock --+ MFBlockStructure
+**Note**: Code generation requires a few extra dependencies, grouped in the `codegen` optional dependency group: `Jinja2` and `modflow-devtools`.
 
-MFData --+ MFDataStructure
+## Input specification
 
-MFData --* MFArray --* MFTransientArray
+The `flopy.mf6.data.mfstructure.MFStructure` class represents an input specification. The class is a singleton, meaning only one instance of this class can be created.  The class contains a sim_struct attribute (which is a flopy.mf6.data.mfstructure.MFSimulationStructure object) which contains all of the meta-data for all package files.  Meta-data is stored in a structured format. MFSimulationStructure contains MFModelStructure and MFInputFileStructure objects, which contain the meta-data for each model type and each "simulation-level" package (tdis, ims, ...).  MFModelStructure contains model specific meta-data and a MFInputFileStructure object for each package in that model.  MFInputFileStructure contains package specific meta-data and a MFBlockStructure object for each block contained in the package file.  MFBlockStructure contains block specific meta-data and a MFDataStructure object for each data structure defined in the block, and MFDataStructure contains data structure specific meta-data and a MFDataItemStructure object for each data item contained in the data structure.  Data structures define the structure of data that is naturally grouped together, for example, the data in a numpy recarray.  Data item structures define the structure of specific pieces of data, for example, a single column of a numpy recarray.  The meta-data defined in these classes provides all the information FloPy needs to read and write MODFLOW 6 package and name files, create the Flopy interface, and check the data for various constraints.
 
-MFData --* MFList --* MFTransientList
+```mermaid
+classDiagram
+    MFStructure *-- "1" MFSimulationStructure : has
+    MFSimulationStructure *-- "1+" MFModelStructure : has
+    MFModelStructure *-- "1" MFInputFileStructure : has
+    MFInputFileStructure *-- "1+" MFBlockStructure : has
+    MFBlockStructure *-- "1+" MFDataStructure : has
+    MFDataStructure *-- "1+" MFDataItemStructure : has
+```
 
-MFData --* MFScalar --* MFTransientScalar
+Figure 1: Generic data structure hierarchy.  Connections show composition relationships.
+
+The package and data classes are related as shown below in figure 2.  On the top of the figure 2 is the MFPackage class, which is the base class for all packages.  MFPackage contains generic methods for building data objects and reading and writing the package to a file.  MFPackage contains a MFInputFileStructure object that defines how the data is structured in the package file.  MFPackage also contains a dictionary of blocks (MFBlock).  The MFBlock class is a generic class used to represent a block within a package.  MFBlock contains a MFBlockStructure object that defines how the data in the block is structured.  MFBlock also contains a dictionary of data objects (subclasses of MFData) contained in the block and a list of block headers (MFBlockHeader) for that block.  Block headers contain the block's name and optionally data items (eg. iprn).
 
-MFTransientData --* MFTransientArray, MFTransientList, MFTransientScalar
+```mermaid
+classDiagram
+
+MFPackage *-- "1+" MFBlock : has
+MFBlock *-- "1+" MFData : has
+MFPackage *-- "1" MFInputFileStructure : has
+MFBlock *-- "1" MFBlockStructure : has
+MFData *-- "1" MFDataStructure : has
+MFData <|-- MFArray
+MFArray <|-- MFTransientArray
+MFData <|-- MFList
+MFList <|-- MFTransientList
+MFData <|-- MFScalar
+MFScalar <|-- MFTransientScalar
+MFTransientData <|-- MFTransientArray
+MFTransientData <|-- MFTransientList
+MFTransientData <|-- MFTransientScalar
+```
 							 
 Figure 2:  FPMF6 package and data classes.  Lines connecting classes show a relationship defined between the two connected classes.  A "*" next to the class means that the  class is a sub-class of the connected class.  A "+" next to the class means that the class is contained within the connected class.
-***
 
 There are three main types of data, MFList, MFArray, and MFScalar data.  All three of these data types are derived from the MFData abstract base class.  MFList data is the type of data stored in a spreadsheet with different column headings.  For example, the data describing a flow barrier are of type MFList.  MFList data is stored in numpy recarrays.  MFArray data is data of a single type (eg. all integer values).  For example, the model's HK values are of type MFArray.  MFArrays are stored in numpy ndarrays.  MFScalar data is a single data item.  Most MFScalar data are options.  All MFData subclasses contain an MFDataStructure object that defines the expected structure and types of the data.
 

etc/environment.yml

@@ -10,6 +10,14 @@ dependencies:
   - matplotlib>=1.4.0
   - pandas>=2.0.0
 
+  # codegen
+  - boltons>=1.0
+  - Jinja2>=3.0
+  - tomli
+  - tomli-w
+  - pip:
+      - git+https://github.com/MODFLOW-ORG/modflow-devtools.git
+
   # lint
   - cffconvert
   - codespell>=2.2.2

flopy/mf6/data/dfn/utl-ts.dfn

@@ -62,6 +62,7 @@ valid stepwise linear linearend
 shape time_series_names
 tagged false
 reader urword
+in_record true
 optional false
 in_record true
 longname
@@ -112,6 +113,7 @@ name sfacs
 type keyword
 shape
 reader urword
+in_record true
 optional false
 in_record true
 longname

flopy/mf6/data/mfdatastorage.py

@@ -316,7 +316,7 @@ def __init__(
         self.data_structure_type = data_structure_type
         package_dim = self.data_dimensions.package_dim
         self.in_model = (
-            self.data_dimensions is not None
+            package_dim is not None
             and len(package_dim.package_path) > 1
             and package_dim.model_dim[0].model_name is not None
             and package_dim.model_dim[0].model_name.lower()

flopy/mf6/data/mfdatautil.py

@@ -1054,89 +1054,3 @@ def empty(
                 return template
         else:
             return rec_array
-
-
-class MFDocString:
-    """
-    Helps build a python class doc string
-
-    Parameters
-    ----------
-    description : string
-        description of the class
-
-    Attributes
-    ----------
-    indent: string
-        indent to use in doc string
-    description : string
-        description of the class
-    parameter_header : string
-        header for parameter section of doc string
-    parameters : list
-        list of docstrings for class parameters
-
-    Methods
-    -------
-    add_parameter : (param_descr : string, beginning_of_list : bool)
-        adds doc string for a parameter with description 'param_descr' to the
-        end of the list unless beginning_of_list is True
-    get_doc_string : () : string
-        builds and returns the docstring for the class
-    """
-
-    def __init__(self, description):
-        self.indent = "    "
-        self.description = description
-        self.parameter_header = (
-            f"{self.indent}Parameters\n{self.indent}----------"
-        )
-        self.parameters = []
-        self.model_parameters = []
-
-    def add_parameter(
-        self, param_descr, beginning_of_list=False, model_parameter=False
-    ):
-        if beginning_of_list:
-            self.parameters.insert(0, param_descr)
-            if model_parameter:
-                self.model_parameters.insert(0, param_descr)
-        else:
-            self.parameters.append(param_descr)
-            if model_parameter:
-                self.model_parameters.append(param_descr)
-
-    def get_doc_string(self, model_doc_string=False, sim_doc_string=False):
-        doc_string = '{}"""\n{}{}\n\n{}\n'.format(
-            self.indent, self.indent, self.description, self.parameter_header
-        )
-        if model_doc_string:
-            param_list = self.model_parameters
-            doc_string = (
-                "{}    modelname : string\n        name of the "
-                "model\n    model_nam_file : string\n"
-                "        relative path to the model name file from "
-                "model working folder\n    version : string\n"
-                "        version of modflow\n    exe_name : string\n"
-                "        model executable name\n"
-                "    model_ws : string\n"
-                "        model working folder path"
-                "\n".format(doc_string)
-            )
-        else:
-            param_list = self.parameters
-        for parameter in param_list:
-            if sim_doc_string:
-                pclean = parameter.strip()
-                if (
-                    pclean.startswith("simulation")
-                    or pclean.startswith("loading_package")
-                    or pclean.startswith("filename")
-                    or pclean.startswith("pname")
-                    or pclean.startswith("parent_file")
-                ):
-                    continue
-            doc_string += f"{parameter}\n"
-        if not (model_doc_string or sim_doc_string):
-            doc_string += f'\n{self.indent}"""'
-        return doc_string