Refactor `Measure` block metrics to be homeserver-scoped #18591

MadLittleMods · 2025-06-23T20:43:39Z

Refactor Measure block metrics to be homeserver-scoped.

Spawning from #18443

Testing strategy

See behavior of previous `metrics` listener

Add the metrics listener in your homeserver.yaml

listeners:
  - port: 9323
    type: metrics
    bind_addresses: ['127.0.0.1']

Start the homeserver: poetry run synapse_homeserver --config-path homeserver.yaml
Fetch http://localhost:9323/metrics
Observe response includes the block metrics (synapse_util_metrics_block_count, synapse_util_metrics_block_in_flight, etc)

See behavior of the `http` `metrics` resource

Add the metrics resource to a new or existing http listeners in your homeserver.yaml

listeners:
  - port: 9322
    type: http
    bind_addresses: ['127.0.0.1']
    resources:
      - names: [metrics]
        compress: false

Start the homeserver: poetry run synapse_homeserver --config-path homeserver.yaml
Fetch http://localhost:9322/_synapse/metrics (it's just a GET request so you can even do in the browser)
Observe response includes the block metrics (synapse_util_metrics_block_count, synapse_util_metrics_block_in_flight, etc)

Dev notes

HomeserverMetricsManager

Todo

Fix the usages of Measure

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

Fix `HomeserverMetricsManager` lints Fix `metrics_manager` missing in `Measure` `__slots__`

Not finished: - `synapse/http/federation/well_known_resolver.py` - `measure_func`

…round Fix mypy complaints ``` synapse/handlers/delayed_events.py:266: error: Cannot determine type of "validator" [has-type] synapse/handlers/delayed_events.py:267: error: Cannot determine type of "event_builder_factory" [has-type] ```

This matches the pattern of other homeserver attributes like `hs.get_clock()`

synapse/metrics/homeserver_metrics_manager.py

…output

MadLittleMods · 2025-06-24T23:33:01Z

synapse/app/_base.py

+def listen_metrics(
+    bind_addresses: StrCollection, port: int, metrics_manager: HomeserverMetricsManager
+) -> None:
    """
    Start Prometheus metrics server.
    """
-    from prometheus_client import start_http_server as start_http_server_prometheus
+    from prometheus_client import (
+        REGISTRY,
+        CollectorRegistry,
+        start_http_server as start_http_server_prometheus,
+    )

-    from synapse.metrics import RegistryProxy
+    from synapse.metrics import CombinedRegistryProxy
+
+    combined_registry_proxy = CombinedRegistryProxy(
+        [
+            # TODO: Remove `REGISTRY` once all metrics have been migrated to the
+            # homeserver specific metrics collector registry, see
+            # https://github.com/element-hq/synapse/issues/18592
+            REGISTRY,
+            metrics_manager.metrics_collector_registry,
+        ]
+    )
+    # Cheeky cast but matches the signature of a `CollectorRegistry` instance enough
+    # for it to be usable in the contexts in which we use it.
+    # TODO Do something nicer about this.
+    registry = cast(CollectorRegistry, combined_registry_proxy)


All of this work can be removed if we decide that it's okay to drop the type: metrics listener (part of #18584)

MadLittleMods · 2025-06-24T23:33:24Z

synapse/metrics/__init__.py

-    @staticmethod
-    def collect() -> Iterable[Metric]:
-        for metric in REGISTRY.collect():
-            if not metric.name.startswith("__"):


I don't see any metrics that start with __ so I've removed this logic

MadLittleMods · 2025-06-24T23:37:11Z

synapse/metrics/__init__.py

-        for metric in REGISTRY.collect():
-            if not metric.name.startswith("__"):
-                yield metric
+class CombinedRegistryProxy:


CombinedRegistryProxy can be removed if we decide that it's okay to drop the type: metrics listener (part of #18584) since it's the only usage.

MadLittleMods · 2025-06-24T23:38:33Z

synapse/metrics/_twisted_exposition.py

+        # While we're in the middle of the refactor of metrics in Synapse, we need to
+        # merge the metrics from the global registry and the homeserver specific metrics
+        # collector registry.
+        #
+        # TODO: Remove `generate_latest(REGISTRY)` once all homeserver metrics have been
+        # migrated to the homeserver specific metrics collector registry, see
+        # https://github.com/element-hq/synapse/issues/18592
+        response = generate_latest(REGISTRY) + generate_latest(
+            self.metrics_manager.metrics_collector_registry
+        )


Instead of using CombinedRegistryProxy here, I've decided to do something way more straight-forward and avoid the cheeky casting.

We have to use CombinedRegistryProxy in the other spot because we don't control how it collects metrics.

MadLittleMods · 2025-06-24T23:39:41Z

synapse/metrics/homeserver_metrics_manager.py

+        self.in_flight: InFlightGauge[BlockInFlightMetric] = InFlightGauge(
+            "synapse_util_metrics_block_in_flight",
+            "",
+            labels=["block_name"],
+            # Matches the fields in the `BlockInFlightMetric`
+            sub_metrics=["real_time_max", "real_time_sum"],
+        )


Currently, this specific metric doesn't use the homeserver metrics_collector_registry. This is planned for a future PR where we update InFlightGauge to support a custom registry option. Tracked in #18592

synapse/metrics/homeserver_metrics_manager.py

MadLittleMods · 2025-06-24T23:46:25Z

synapse/app/_base.py

+    # Cheeky cast but matches the signature of a `CollectorRegistry` instance enough
+    # for it to be usable in the contexts in which we use it.
+    # TODO Do something nicer about this.
+    registry = cast(CollectorRegistry, combined_registry_proxy)


This cast is the same thing we were doing before with the RegistryProxy, nothing new. I'm not very keen on thinking of something better here.

…mock

MadLittleMods · 2025-06-26T21:14:37Z

Per our decision in #18592 (comment) to add instance labels to the metrics directly instead of using a separate reigstry, closing in favor of #18601

Refactor block metrics to use HS specific registry

676a2cc

Fix `HomeserverMetricsManager` lints Fix `metrics_manager` missing in `Measure` `__slots__`

MadLittleMods added the A-Metrics label Jun 23, 2025

MadLittleMods mentioned this pull request Jun 23, 2025

Refactor metrics to be scoped to the homeserver #18592

Open

25 tasks

MadLittleMods added 7 commits June 24, 2025 16:14

Use global REGISTRY for now

420c83d

Refactor InFlight block metrics to be HS specific

612cbd9

Add docstrings

c07e1f5

Merge branch 'develop' into madlittlemods/per-hs-metrics-measure

96c9485

Refactor bulk Measure usage

019f0b7

Not finished: - `synapse/http/federation/well_known_resolver.py` - `measure_func`

Refactor Measure in synapse/http/federation/well_known_resolver.py

c5296f5

Update measure_func to support HasClockAndMetricsManager

41dfb20

MadLittleMods force-pushed the madlittlemods/per-hs-metrics-measure branch 2 times, most recently from 1962fc8 to 5c5090f Compare June 24, 2025 21:34

MadLittleMods added 3 commits June 24, 2025 16:35

Make sure @measure_func usages have the necessary properties

f469042

Add changelog

e46690c

MadLittleMods force-pushed the madlittlemods/per-hs-metrics-measure branch from 5c5090f to e46690c Compare June 24, 2025 21:35

Refactor from hs.metrics_manager -> hs.get_metrics_manager()

3277afd

This matches the pattern of other homeserver attributes like `hs.get_clock()`

MadLittleMods changed the title ~~Refactor Measure block metrics to be homeserver-scoped~~ Refactor Measure block metrics to be be ready to be homeserver-scoped Jun 24, 2025

MadLittleMods commented Jun 24, 2025

View reviewed changes

synapse/metrics/homeserver_metrics_manager.py Outdated Show resolved Hide resolved

MadLittleMods added 2 commits June 24, 2025 16:50

Update doc comment further

ee51a45

Use homeserver specific registry and combine in the metrics endpoint …

554673d

…output

MadLittleMods commented Jun 24, 2025

View reviewed changes

Better docstring

f6f079b

MadLittleMods changed the title ~~Refactor Measure block metrics to be be ready to be homeserver-scoped~~ Refactor Measure block metrics to be homeserver-scoped Jun 24, 2025

MadLittleMods commented Jun 24, 2025

View reviewed changes

MadLittleMods added 2 commits June 24, 2025 18:51

Fix tests.test_state missing get_metrics_manager() on homeserver …

71f415d

…mock

Fill in get_metrics_manager() on port db MockHomeserver

1a5e9f6

MadLittleMods marked this pull request as ready for review June 25, 2025 01:30

MadLittleMods requested a review from a team as a code owner June 25, 2025 01:30

MadLittleMods marked this pull request as draft June 25, 2025 18:22

MadLittleMods mentioned this pull request Jun 26, 2025

Refactor Measure block metrics to be homeserver-scoped (v2) #18601

Open

3 tasks

MadLittleMods closed this Jun 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor `Measure` block metrics to be homeserver-scoped #18591

Refactor `Measure` block metrics to be homeserver-scoped #18591

Uh oh!

MadLittleMods commented Jun 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

MadLittleMods Jun 24, 2025 •

edited

Loading

Uh oh!

MadLittleMods Jun 24, 2025

Uh oh!

MadLittleMods Jun 24, 2025

Uh oh!

MadLittleMods Jun 24, 2025

Uh oh!

MadLittleMods Jun 24, 2025

Uh oh!

Uh oh!

MadLittleMods Jun 24, 2025

Uh oh!

MadLittleMods commented Jun 26, 2025

Uh oh!

Uh oh!

Refactor Measure block metrics to be homeserver-scoped #18591

Refactor Measure block metrics to be homeserver-scoped #18591

Uh oh!

Conversation

MadLittleMods commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing strategy

See behavior of previous metrics listener

See behavior of the http metrics resource

Dev notes

Todo

Pull Request Checklist

Uh oh!

Uh oh!

MadLittleMods Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MadLittleMods Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods commented Jun 26, 2025

Uh oh!

Uh oh!

Refactor `Measure` block metrics to be homeserver-scoped #18591

Refactor `Measure` block metrics to be homeserver-scoped #18591

MadLittleMods commented Jun 23, 2025 •

edited

Loading

See behavior of previous `metrics` listener

See behavior of the `http` `metrics` resource

MadLittleMods Jun 24, 2025 •

edited

Loading