Add strict modules docs

Reviewed By: carljm

Differential Revision: D35038739

fbshipit-source-id: 27fed27

Author: itamaro

CinderDoc/strict_modules/DOCS

@@ -0,0 +1,6 @@
+[strict_modules]
+  type = WIKI
+  srcs = [
+      glob(*.rst)
+    ]
+  wiki_root_path = Python/Cinder/External_Public/Strict_Modules

CinderDoc/strict_modules/guide/conversion/class_inst_conflict.rst

@@ -0,0 +1,48 @@
+Class / Instance Conflict
+#########################
+
+
+One of the changes that strict modules introduces is the promotion of instance
+members to being class level declarations.  For more information on this pattern
+see :doc:`../limitations/class_attrs`.
+
+A typical case for this is when you'd like to have a default method implementation
+but override it on a per instance basis:
+
+.. code-block:: python
+
+    class C:
+        def f(self):
+            return 42
+
+    a = C()
+    a.f = lambda: "I'm a special snowflake"
+
+
+If you attempt this inside of a strict module you'll get an AttributeError that
+says "'C' object attribute 'f' is read-only".  This is because the instance
+doesn't have any place to store the method.  You might think that you can declare
+the field explicitly as specified in the documentation:
+
+.. code-block:: python
+
+    class C:
+        f: ...
+        def f(self):
+            return 42
+
+But instead you'll get an error reported by strict modules stating that there's
+a conflict with the variable.  To get around this issue you can promote the function
+to always be treated as an instance member:
+
+.. code-block:: python
+
+    class C:
+        def __init__(self):
+            self.f = self.default_f
+
+        def default_f(self):
+            return 42
+
+    a = C()
+    a.f = lambda: "I'm a special snowflake" # Ok, you are a special snowflake

CinderDoc/strict_modules/guide/conversion/external_modification.rst

@@ -0,0 +1,53 @@
+Modifying External State
+########################
+
+Strict modules enforces object :doc:`ownership </guide/limitations/ownership>`,
+and will not allow module-level code to modify any object defined
+in a different module.
+
+One common example of this is to have a global registry of some sort of
+objects:
+
+**methods.py**
+
+.. code-block:: python
+
+    import __strict__
+
+    ROUTES = list()
+
+    def route(f):
+        ROUTES.append(f)
+        return f
+
+**routes.py**
+
+.. code-block:: python
+
+    import __strict__
+
+    from methods import route
+
+    @route
+    def create_user(*args):
+        ...
+
+
+Here we have one module which is maintaining a global registry, which is
+populated as a side effect of importing another module.  If for some reason
+one module doesn't get imported or if the order of imports changes then the
+program's execution can change. When strict modules analyzes this code it will
+report a :doc:`/guide/errors/modify_imported_value`.
+
+A better pattern for this is to explicitly register the values in a central
+location:
+
+**methods.py**
+
+.. code-block:: python
+
+    import __strict__
+
+    from routes import create_user
+
+    ROUTES = [create_user, ...]

CinderDoc/strict_modules/guide/conversion/index.rst

@@ -0,0 +1,12 @@
+Conversion Tips
+###############
+
+This section of the documentation includes common patterns that violate the
+limitations of strict modules and solutions you can use to work around them.
+
+.. toctree::
+    :maxdepth: 1
+    :titlesonly:
+    :glob:
+
+    *

CinderDoc/strict_modules/guide/conversion/loose_slots.rst

@@ -0,0 +1,42 @@
+The @loose_slots decorator
+##########################
+
+Instances of strict classes have `__slots__
+<https://docs.python.org/3/reference/datamodel.html#slots>`_ automatically
+created for them. This means they will raise ``AttributeError`` if you try to
+add any attribute to them that isn't declared with a type annotation on the
+class itself (e.g. ``attrname: int``) or assigned in the ``__init__`` method.
+
+When initially converting a module to strict, if it is widely-used it can be
+hard to verify that there isn't code somewhere tacking extra attributes onto
+instances of classes defined in that module. In this case, you can temporarily
+place the ``strict_modules.loose_slots`` decorator on the class for a safer
+transition. Example:
+
+.. code-block:: python
+
+    import __strict__
+
+    from compiler.strict.runtime import loose_slots
+
+    @loose_slots
+    class MyClass:
+        ...
+
+This decorator will allow extra attributes to be added to the class, but will
+fire a warning when it happens. You can access these warnings by setting a
+warnings callback function:
+
+.. code-block:: python
+
+    from cinder import cinder_set_warnings_handler
+
+    def log_cinder_warning(msg: str, *args: object) -> None:
+        # ...
+
+    cinder_set_warnings_handler(log_cinder_warning)
+
+Typically you'd want to set a warnings handler that logs these warnings somewhere,
+then you can deploy some new strict modules using `@loose_slots`,
+and once the code has been in production for a bit and you see no warnings
+fired, you can safely remove `@loose_slots`.

CinderDoc/strict_modules/guide/conversion/module_access.rst

@@ -0,0 +1,27 @@
+Top-level Module Access
+#######################
+
+A common pattern is to import a module and access members from that module:
+
+.. code-block:: python
+
+    from useful import submodule
+
+    class MyClass(submodule.BaseClass):
+        pass
+
+If “submodule” is not strict, then we don't know what it is and what side
+effects could happen by dotting through it. So this pattern is disallowed
+inside of a strict module when importing from a non-strict module. Instead
+you can transform the code to:
+
+.. code-block:: python
+
+    from useful.submodule import BaseClass
+
+    class MyClass(BaseClass):
+        pass
+
+This will cause any side effects that are possible to occur only when
+the non-strict module is imported; the execution of the rest of the
+strict module will be known to be side effect free.

CinderDoc/strict_modules/guide/conversion/singletons.rst

@@ -0,0 +1,66 @@
+Global Singletons
+#################
+
+Sometimes it might be useful to encapsulate a set of functionality into
+a class and then have a global singleton of that class. And sometimes
+that global singleton might have dependencies on non-strict code which
+makes it impossible to construct at the top-level in a strict module.
+
+.. code-block:: python
+
+    from non_strict import get_counter_start
+
+    class Counter:
+        def __init__(self) -> None:
+            self.value: int = get_counter_start()
+
+        def next(self) -> int:
+            res = self.value
+            self.value += 1
+            return res
+
+    COUNTER = Counter()
+
+One way to address this is to refactor the Counter class so that it
+does less when constructed, delaying some work until first use. For
+example:
+
+.. code-block:: python
+
+    from non_strict import get_counter_start
+
+    class Counter:
+        def __init__(self) -> None:
+            self.value: int = -1
+
+        def next(self) -> int:
+            if self.value == -1:
+                self.value = get_counter_start()
+            res = self.value
+            self.value += 1
+            return res
+    COUNTER = Counter()
+
+Another approach is that instead of constructing the singleton at the
+top of the file you can push this into a function so it gets defined
+the first time it'll need to be used:
+
+.. code-block:: python
+
+    _COUNTER = None
+
+    def get_counter() -> Counter:
+        global _COUNTER
+        if _COUNTER is None:
+            _COUNTER = Counter()
+        return _COUNTER
+
+You can also use an lru_cache instead of a global variable:
+
+.. code-block:: python
+
+    from functools import lru_cache
+
+    @lru_cache(maxsize=1)
+    def get_counter() -> Counter:
+        return Counter()

CinderDoc/strict_modules/guide/conversion/splitting_modules.rst

@@ -0,0 +1,69 @@
+Splitting Modules
+#################
+
+Sometimes a module might contain functionality which is dependent upon certain
+behavior which cannot be analyzed - either it truly has external side effects,
+it is dependent upon another module which cannot yet be strictified and needs
+to be used at the top-level, or it is dependent upon something which strict
+modules have not yet been able to analyze.
+
+In these cases one possible solution, although generally a last resort,
+is to break the module into two modules. The first module will only contain
+the code which cannot be safely strictified.  The second module will contain
+all of the code that can be safely treated as strict.  A better way to do this
+is to not have the unverifable code happen at startup, but if that's not
+possible then splitting is an acceptable option.
+
+Because strict modules can still import non-strict modules the strict module
+can continue to expose the same interface as it previously did, and no other
+code needs to be updated.  The only limitation to this is that it requires
+that the module being strictified doesn't need to interact with the non-strict
+elements at the top level.  For example classes could still create instances
+of them, but the strict module couldn't call functions in the non-strict
+module at the top level.
+
+
+.. code-block:: python
+
+    import csv
+    from random import choice
+
+    FAMOUS_PEOPLE = list(csv.reader(open('famous_people.txt').readlines()))
+
+    class FamousPerson:
+        def __init__(self, name, age, height):
+                self.name = name
+                self.age = int(age)
+                self.height = float(height)
+
+    def get_random_person():
+        return FamousPerson(*choice(FAMOUS_PEOPLE))
+
+
+We can split this into two modules, one which does the unverifable read of our
+sample data from disk and another which returns the random piece of sample data:
+
+
+.. code-block:: python
+
+    import csv
+
+    FAMOUS_PEOPLE = list(csv.reader(open('famous_people.txt').readlines()))
+
+
+And we can have another module which exports our FamousPerson class along with
+the API to return a random famous person:
+
+.. code-block:: python
+
+    from random import choice
+    from famous_people_data import FAMOUS_PEOPLE
+
+    class FamousPerson:
+        def __init__(self, name, age, height):
+                self.name = name
+                self.age = int(age)
+                self.height = float(height)
+
+    def get_random_person():
+        return FamousPerson(*choice(FAMOUS_PEOPLE))

CinderDoc/strict_modules/guide/conversion/stubs.rst

@@ -0,0 +1,67 @@
+Strict Module Stubs
+###################
+
+Sometimes your modules depend on other modules that cannot be directly
+strictified - it could depend on a Cython module, or a module from a
+third-party library whose source code you can't modify.
+
+In this situation, if you are certain that the dependency is strict, you
+can provide a strict module stub file (`.pys`) describing the behavior of
+the module. Put the strict module stub file in your strict module stubs directory
+(this is configured via `-X strict-module-stubs-path=...=` or
+`PYTHONSTRICTMODULESTUBSPATH` env var, or by subclassing `StrictSourceFileLoader`
+and passing a `stub_path` argument to `super().__init__(...)`.)
+
+There are two ways to stub a class or function in a strict module stub file.
+You can provide a full Python implementation, which is useful in the case
+of stubbing a Cython file, or you can just provide a function/class name,
+with a `@implicit` decorator. In the latter case, the stub triggers the
+strict module analyzer to look for the source code on `sys.path` and analyze
+the source code.
+
+If the module you depend on is already actually strict-compliant you can
+simplify the stub file down to just contain the single line `__implicit__`,
+which just says "go use the real module contents, they're fine".
+See `Lib/compiler/strict/stubs/_collections_abc.pys` for an existing example.
+Per-class/function stubs are only needed where the stdlib module does
+non-strict things at module level, so we need to extract just the bits we
+depend on and verify them for strictness.
+
+If both a `.py` file and a `.pys` file exist, the strict module analyzer will
+prioritize the `.pys` file. This means adding stubs to existing
+modules in your codebase will shadow the actual implementation.
+You should probably avoid doing this.
+
+Example of Cython stub:
+
+**myproject/worker.py**
+
+.. code-block:: python
+
+    from some_cython_mod import plus1
+
+    two = plus1(1)
+
+
+Here you can provide a stub for the Cython implementation of `plus1`
+
+**strict_modules/stubs/some_cython_mod.pys**
+
+.. code-block:: python
+
+    # a full reimplementation of plus1
+    def plus1(arg):
+        return arg + 1
+
+Suppose you would like to use the standard library functions `functools.wraps`,
+but the strict module analysis does not know of the library. You can add an implicit
+stub:
+
+**strict_modules/stubs/functools.pys**
+
+.. code-block:: python
+
+    @implicit
+    def wraps(): ...
+
+You can mix explicit and implicit stubs. See `Lib/compiler/strict/stubs` for some examples.