Standardise the module interface

CHANGES.md

@@ -1,3 +1,11 @@
+Unreleased
+==========
+
+The spam checker interface is now deprecated in favour of a new generic module system and
+will be removed in Synapse 1.40 (planned for mid-August 2021). See the
+[upgrade notes](https://github.com/matrix-org/synapse/blob/develop/UPGRADE.rst#deprecation-of-the-current-spam-checker-interface)
+for more info.
+
 Synapse 1.36.0 (2021-06-15)
 ===========================
 

UPGRADE.rst

@@ -85,6 +85,24 @@ for example:
      wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
      dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
 
+Upgrading to v1.37.0
+====================
+
+Deprecation of the current spam checker interface
+-------------------------------------------------
+
+The current spam checker interface is deprecated in favour of a new generic modules system.
+Authors of spam checker modules can refer to `this documentation <https://matrix-org.github.io/synapse/develop/modules.html#porting-an-existing-module-that-uses-the-old-interface>`_
+to update their modules. Synapse administrators can refer to `this documentation <https://matrix-org.github.io/synapse/develop/modules.html#using-modules>`_
+to update their configuration once the modules they are using have been updated.
+
+Support for the current spam checker interface will be removed in Synapse 1.40 (planned for
+mid-August 2021).
+
+More module interfaces will be ported over to this new generic system in future versions
+of Synapse.
+
+
 Upgrading to v1.34.0
 ====================
 

docs/modules.md

@@ -1,6 +1,6 @@
 # Modules
 
-Synapse supports extending its functionalities by configuring external modules.
+Synapse supports extending its functionality by configuring external modules.
 
 ## Using modules
 
@@ -23,13 +23,28 @@ custom code on your Synapse homeserver. Server admins are encouraged to verify t
 provenance of the modules they use on their homeserver and make sure the modules aren't
 running malicious code on their instance.
 
+Also note that we are currently in the process of migrating module interfaces to this
+system. While some interfaces might be compatible with it, others still require
+configuring modules in another part of Synapse's configuration file. Currently, only the
+spam checker interface is compatible with this new system.
+
 ## Writing a module
 
 A module is a Python class that uses Synapse's module API to interact with the
 homeserver. It can register callbacks that Synapse will call on specific operations, as
 well as web resources to attach to Synapse's web server.
 
-A module must implement the following static method:
+When instantiated, a module is given its parsed configuration as well as an instance of
+the `synapse.module_api.ModuleApi` class. The configuration is a dictionary, and is
+either the output of the module's `parse_config` static method (see below), or the
+configuration associated with the module in Synapse's configuration file.
+
+See the documentation for the `ModuleApi` class
+[here](https://github.com/matrix-org/synapse/blob/master/synapse/module_api/__init__.py).
+
+### Handling the module's configuration
+
+A module can implement the following static method:
 
 ```python
 @staticmethod
@@ -38,13 +53,9 @@ def parse_config(config: dict) -> dict
 
 This method is given a dictionary resulting from parsing the YAML configuration for the
 module. It may modify it (for example by parsing durations expressed as strings (e.g.
-"5d") into milliseconds, etc.), and return the modified dictionary. If no change is
-necessary, this method should just return `config`.
-
-When instantiated, a module is given its parsed configuration (i.e. the output of
-`parse_config`) as well as an instance of the `synapse.module_api.ModuleApi` class.
-
-See the documentation for the `ModuleApi` class [here](/synapse/module_api/__init__.py).
+"5d") into milliseconds, etc.), and return the modified dictionary. It may also verify
+that the configuration is correct, and raise an instance of
+`synapse.module_api.errors.ConfigError` if not.
 
 ### Registering a web resource
 
@@ -75,10 +86,10 @@ Modules **must** register their web resources in their `__init__` method.
 ### Registering a callback
 
 Modules can use Synapse's module API to register callbacks. Callbacks are functions that
-Synapse will call when performing specific actions. Callbacks can be either asynchronous
-or synchronous, and are split in categories. A single module may implement callbacks from
-multiple categories, and is under no obligation to implement all callbacks from the
-category(ies) it registers callbacks for.
+Synapse will call when performing specific actions. Callbacks must be asynchronous, and
+are split in categories. A single module may implement callbacks from multiple categories,
+and is under no obligation to implement all callbacks from the categories it registers
+callbacks for.
 
 #### Spam checker callbacks
 
@@ -177,6 +188,24 @@ def check_media_file_for_spam(
 Called when storing a local or remote file. The module must return a boolean indicating
 whether the given file can be stored in the homeserver's media store.
 
+### Porting an existing module that uses the old interface
+
+In order to port a module that uses Synapse's old module interface, its author needs to:
+
+* ensure the module's callbacks are all asynchronous.
+* register their callbacks using one or more of the `register_[...]_callbacks` methods
+  from the `ModuleApi` class in the module's `__init__` method (see [this section](#registering-a-web-resource)
+  for more info).
+
+Additionally, if the module is packaged with an additional web resource, the module
+should register this resource in its `__init__` method using the `register_web_resource`
+method from the `ModuleApi` class (see [this section](#registering-a-web-resource) for
+more info).
+
+The module's author should also update any example in the module's configuration to only
+use the new `modules` section in Synapse's configuration file (see [this section](#using-modules)
+for more info).
+
 ### Example
 
 The example below is a module that implements the spam checker callback
@@ -197,7 +226,7 @@ class DemoResource(Resource):
         super(DemoResource, self).__init__()
         self.config = config
 
-    async def render_GET(self, request: Request):
+    def render_GET(self, request: Request):
         name = request.args.get(b"name")[0]
         request.setHeader(b"Content-Type", b"application/json")
         return json.dumps({"hello": name})

docs/sample_config.yaml

@@ -34,8 +34,7 @@
 
 ## Modules ##
 
-# Server admins can expand Synapse's functionalities by using external modules
-# to complement certain operations.
+# Server admins can expand Synapse's functionality with external modules.
 #
 # See https://matrix-org.github.io/synapse/develop/modules.html for more
 # documentation on how to configure or create custom modules for Synapse.
@@ -2586,19 +2585,6 @@ push:
   #group_unread_count_by_room: false
 
 
-# Spam checkers are third-party modules that can block specific actions
-# of local users, such as creating rooms and registering undesirable
-# usernames, as well as remote users by redacting incoming events.
-#
-spam_checker:
-   #- module: "my_custom_project.SuperSpamChecker"
-   #  config:
-   #    example_option: 'things'
-   #- module: "some_other_project.BadEventStopper"
-   #  config:
-   #    example_stop_events_from: ['@bad:example.com']
-
-
 ## Rooms ##
 
 # Controls whether locally-created rooms should be end-to-end encrypted by

synapse/app/generic_worker.py

@@ -354,6 +354,10 @@ def _listen_http(self, listener_config: ListenerConfig):
                 if name == "replication":
                     resources[REPLICATION_PREFIX] = ReplicationRestResource(self)
 
+        # Attach additional resources registered by modules.
+        resources.update(self._module_web_resources)
+        self._module_web_resources_consumed = True
+
         root_resource = create_resource_tree(resources, OptionsResource())
 
         _base.listen_tcp(

synapse/config/modules.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+from typing import Any, Dict, List, Tuple
 
 from synapse.config._base import Config, ConfigError
 from synapse.util.module_loader import load_module
@@ -20,7 +21,7 @@ class ModulesConfig(Config):
     section = "modules"
 
     def read_config(self, config: dict, **kwargs):
-        self.loaded_modules = []
+        self.loaded_modules: List[Tuple[Any, Dict]] = []
 
         configured_modules = config.get("modules") or []
         for i, module in enumerate(configured_modules):
@@ -34,8 +35,7 @@ def generate_config_section(self, **kwargs):
         return """
             ## Modules ##
 
-            # Server admins can expand Synapse's functionalities by using external modules
-            # to complement certain operations.
+            # Server admins can expand Synapse's functionality with external modules.
             #
             # See https://matrix-org.github.io/synapse/develop/modules.html for more
             # documentation on how to configure or create custom modules for Synapse.

synapse/config/spam_checker.py

@@ -42,18 +42,3 @@ def read_config(self, config, **kwargs):
                 self.spam_checkers.append(load_module(spam_checker, config_path))
         else:
             raise ConfigError("spam_checker syntax is incorrect")
-
-    def generate_config_section(self, **kwargs):
-        return """\
-        # Spam checkers are third-party modules that can block specific actions
-        # of local users, such as creating rooms and registering undesirable
-        # usernames, as well as remote users by redacting incoming events.
-        #
-        spam_checker:
-           #- module: "my_custom_project.SuperSpamChecker"
-           #  config:
-           #    example_option: 'things'
-           #- module: "some_other_project.BadEventStopper"
-           #  config:
-           #    example_stop_events_from: ['@bad:example.com']
-        """