Turn the persistence into multiple pages and...

...add some clarifications and cross references.

...make `immer-persist` be `immer::persist`.

Author: arximboldi

doc/index.rst

@@ -17,13 +17,21 @@ Contents
    utilities
    memory
 
+.. toctree::
+   :caption: Persistence
+   :maxdepth: 3
+
+   persist/introduction
+   persist/serialization
+   persist/transformation
+   persist/reference
+
 .. toctree::
    :caption: Experimental
    :maxdepth: 3
 
    python
    guile
-   persist
 
 ----
 

doc/persist/introduction.rst

@@ -0,0 +1,89 @@
+Introduction
+===============
+
+The ``immer::persist`` library persists persistent data structures,
+allowing the preservation structural sharing of ``immer`` containers
+when serializing, deserializing or transforming the data.
+
+
+.. warning:: This library is still experimental and it's API may
+   change in the future. The headers can be found in
+   ``immer/extra/persist/...`` and the ``extra`` subpath will be
+   removed once it's interface stabilises.
+
+Dependencies
+------------
+
+In addition to the `dependencies <introduction.html#dependencies>`_ of
+``immer``, this library makes use of **C++20**, `Boost.Hana
+<https://boostorg.github.io/hana/>`_, `fmt <https://fmt.dev/>`_ and
+`cereal <https://uscilab.github.io/cereal/>`_.
+
+Why?
+---------
+
+Preserving structural sharing on disk
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Structural sharing* allows ``immer`` containers to be efficient. At
+runtime, two distinct containers can be operated on independently but
+internally they share nodes and use memory efficiently in that
+way.
+
+However when such containers are serialized in a trival form, for
+example, as JSON lists, this sharing is lost: they become truly
+independent---the same data is stored multiple times on disk, and
+later, when read back into memory, the program has lost the structural
+sharing.
+
+This library operates on the internal structure of ``immer``
+containers: allowing it to be serialized, deserialized and
+transformed. This enables more efficient storage, particularly when
+many nodes are reused, and, even more importantly, preserving
+structural sharing after deserializing the containers.
+
+
+Transforming data with structural sharing
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Consider this scenario where you have multiple
+``immer::vector<std::string>``, where the various instances are
+derived from one another. Some of these vectors would be completely
+identical, while others would have just a few elements different. This
+scenario is not uncommon, for example, when `implement the undo
+history of an application by preseriving the previous
+states <https://sinusoid.es/lager/modularity.html#genericity>`_.
+
+The goal is to apply a transformation function to these vectors with
+something like ``std::transform``.
+
+A direct approach would be to take each vector and create a new vector
+by applying the transformation function for each element. However,
+after this process, all the structural sharing of the original
+containers would be lost---the result would be multiple independent
+vectors without any structural sharing, and the transformation may
+have been applied unnecessarily multiple times to identical elements
+that were previously shared.
+
+This library enables the application of the transformation function
+directly on the nodes, preserving structural sharing. Additionally,
+regardless of how many times a node is reused, the transformation
+needs to be performed only once.
+
+.. _pools:
+How?
+------
+
+To solve this problem, this library introduces the notion of a *pool*.
+
+A **pool** represents a *set* of ``immer`` containers of a specific
+type. For example, we may have a pool that contains all
+``immer::vector<int>`` of our document. You can think of it as a small
+database of ``immer`` containers. When serializing the pool, the
+internal structure of all those ``immer`` containers is written as
+whole, preserving the structural sharing between those containers.
+
+Note that for the most part, the user of the library is not concerned
+with pools, as they are generated automatically from your
+data-structures.  However, you may become aware of them in the JSON
+output or when transforming recursive data structures.

doc/persist/reference.rst

@@ -0,0 +1,36 @@
+Reference
+=========
+
+These are the interfaces of the various parts of the
+``immer::persist`` library.
+
+Policy
+------
+
+.. doxygengroup:: Persist-policy
+   :project: immer
+   :content-only:
+
+
+API Overview
+------------
+
+.. doxygengroup:: persist-api
+   :project: immer
+   :content-only:
+
+
+Transform API
+---------------
+
+.. doxygengroup:: Persist-transform
+   :project: immer
+   :content-only:
+
+
+Exceptions
+----------
+
+.. doxygengroup:: Persist-exceptions
+   :project: immer
+   :content-only:

doc/persist/serialization.rst

@@ -0,0 +1,157 @@
+Serialization
+=========
+
+Serializing your data structures using ``immer::persist`` allows you
+preserve the *structural sharing* across sessions of your application.
+
+This has multiple practical use cases, like storing the undo history
+or the clipboard of a complex application, or applying advanced
+logging techniques.
+
+The library serializes multiple containers together via the notion of
+a :ref:`pool<pools>`.  These pools are produced automatically and
+represent in the JSON the internal structure (trees) that implement
+the Immer containers.
+
+Example
+-------
+.. _first-example:
+
+For this example, we'll use a ``document`` type that contains two
+``immer`` vectors.
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: intro/start-types
+   :end-before:  intro/end-types
+
+Let's say we have two vectors ``v1`` and ``v2``, where ``v2`` is
+derived from ``v1`` so that it shares data with it:
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: intro/start-prepare-value
+   :end-before:  intro/end-prepare-value
+
+We can serialize the document using ``cereal`` with this:
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: intro/start-serialize-with-cereal
+   :end-before:  intro/end-serialize-with-cereal
+
+Generating a JSON like this one:
+
+.. code-block:: c++
+
+   {"value0": {"ints": [1, 2, 3], "ints2": [1, 2, 3, 4, 5, 6]}}
+
+As you can see, ``ints`` and ``ints2`` contain the full linearization
+of each vector.  The structural sharing between these two data
+structures is not represented in its serialized form.
+
+Using pools
+-----------
+
+First, let's make the ``document`` struct compatible with
+``boost::hana``. This way, the ``persist`` library can automatically
+determine what :ref:`pool<pools>` types are needed, and to name the
+pools.
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: intro/start-adapt-document-for-hana
+   :end-before:  intro/end-adapt-document-for-hana
+
+Then using ``immer::persist`` we can serialize it with:
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: intro/start-serialize-with-persist
+   :end-before:  intro/end-serialize-with-persist
+
+Which generates some JSON like this:
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: include:intro/start-persist-json
+   :end-before:  include:intro/end-persist-json
+
+As you can see, the value is serialized with every ``immer`` container
+replaced by an identifier.  This identifier is a key into a
+:ref:`pool<pools>`, which is serialized just after.
+
+.. note::
+   Currently, ``immer-persist`` makes a distiction between
+   pools used for saving containers (*output* pools) and for loading
+   containers (*input* pools), similar to ``cereal`` with its
+   ``InputArchive`` and ``OutputArchive`` distiction.
+
+Currently, ``immer-persist`` focuses on JSON as the serialization
+format and uses the ``cereal`` library internally. In principle, other
+formats and serialization libraries could be supported in the future.
+sharing across sessions.
+
+You can see in the out that the nodes of the trees that make up the
+``immer`` containers are directly represented in the JSON and, because
+we are representing all the containers as a whole, those nodes that
+are referenced in multiple trees can be stored only once. That same
+structure is preserved when reading the pool back from disk and
+reconstructing the vectors (and other containers) from it, thus
+allowing us to preserve the structural sharing across sessions.
+
+Custom policies
+----------
+
+We can use policy to control the names of the pools for each container.
+
+For this example, let's define a new document type ``doc_2``. It will
+also contain another type ``extra_data`` with a ``vector`` of
+``strings`` in it. To demonstrate the responsibilities of the policy,
+the ``doc_2`` type will not be a ``boost::hana::Struct`` and will not
+allow for compile-time reflection.
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: include:start-doc_2-type
+   :end-before:  include:end-doc_2-type
+
+We define the ``doc_2_policy`` as following:
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: include:start-doc_2_policy
+   :end-before:  include:end-doc_2_policy
+
+The ``get_pool_types`` function returns the types of containers that
+should be serialized with pools, in this case it's both ``vector`` of
+``ints`` and ``strings``. The ``save`` and ``load`` functions control
+the name of the document node, in this case it is ``doc2_value``. And
+the ``get_pool_name`` overloaded functions supply the name of the pool
+for each corresponding ``immer`` container. To create and serialize a
+value of ``doc_2``, you can use the following approach:
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: include:start-doc_2-cereal_save_with_pools
+   :end-before:  include:end-doc_2-cereal_save_with_pools
+
+The serialized JSON looks like this:
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: include:start-doc_2-json
+   :end-before:  include:end-doc_2-json
+
+And it can also be loaded from JSON like this:
+
+.. literalinclude:: ../../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: include:start-doc_2-load
+   :end-before:  include:end-doc_2-load
+
+This example also demonstrates a scenario in which the main document
+type ``doc_2`` contains another type ``extra_data`` with a
+``vector``. As you can see in the resulting JSON, nested types are
+also serialized with pools: ``"extra": {"comments": 1}``. Only the ID
+of the ``comments`` ``vector`` is serialized instead of its content.