Skip to content
This repository was archived by the owner on Apr 26, 2024. It is now read-only.

Fix Prometheus metrics being negative (mixed up start/end) #13584

Merged
merged 3 commits into from
Aug 23, 2022
Merged

Fix Prometheus metrics being negative (mixed up start/end) #13584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f7ce80a
Fix prometheus metrics being negative (mixed up start/end)
MadLittleMods Aug 22, 2022
4286012
Add some more fidelity to the metrics
MadLittleMods Aug 22, 2022
1a279bc
Same changelog as #13535
MadLittleMods Aug 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/13584.misc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`).
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since #13535 hasn't shipped in a release, using the same changelog so it merges.

7 changes: 6 additions & 1 deletion synapse/handlers/federation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,14 @@
"sec",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the changes look good, please merge. I'd like to get some baseline data on matrix.org.

[],
buckets=(
0.1,
0.5,
1.0,
2.5,
5.0,
7.5,
10.0,
15.0,
Copy link
Contributor Author

@MadLittleMods MadLittleMods Aug 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Increasing the fidelity of the metrics while we're here.

Ref: #13533 (comment)

20.0,
30.0,
40.0,
Expand Down Expand Up @@ -482,7 +487,7 @@ async def try_backfill(domains: List[str]) -> bool:

processing_end_time = self.clock.time_msec()
backfill_processing_before_timer.observe(
(processing_start_time - processing_end_time) / 1000
(processing_end_time - processing_start_time) / 1000
Copy link
Contributor Author

@MadLittleMods MadLittleMods Aug 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the order of these so we no longer give negative times.

end - start is the correct way

ex. 1200 - 1000 -> 200 (positive) 🎉

)

success = await try_backfill(likely_domains)
Expand Down
10 changes: 10 additions & 0 deletions synapse/handlers/federation_event.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,15 +104,25 @@
"sec",
[],
buckets=(
0.1,
0.25,
0.5,
1.0,
2.5,
5.0,
7.5,
10.0,
15.0,
20.0,
25.0,
30.0,
40.0,
50.0,
60.0,
80.0,
100.0,
120.0,
150.0,
180.0,
"+Inf",
),
Expand Down
6 changes: 5 additions & 1 deletion synapse/rest/client/room.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,9 +116,13 @@ def from_member_count(member_count: int) -> "_RoomSize":
2.5,
5.0,
10.0,
20.0,
30.0,
60.0,
80.0,
100.0,
120.0,
150.0,
180.0,
"+Inf",
),
Expand Down Expand Up @@ -674,7 +678,7 @@ async def on_GET(
room_member_count = await make_deferred_yieldable(room_member_count_deferred)
messsages_response_timer.labels(
room_size=_RoomSize.from_member_count(room_member_count)
).observe((processing_start_time - processing_end_time) / 1000)
).observe((processing_end_time - processing_start_time) / 1000)

return 200, msgs

Expand Down