First cut at metrics for alertmanager sharding operation. #4149
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gotjosh
approved these changes
Apr 30, 2021
pracucci
reviewed
Apr 30, 2021
ranton256
reviewed
May 3, 2021
|
Thanks all for the reviews and good feedback, appreciated as always. Will fix up now. |
This commit adds a number of metrics to help track the operation of the alertmanager sharding, specifically around the handling of the state. - `cortex_alertmanager_fetch_replica_state_total` - `cortex_alertmanager_fetch_replica_state_failed_total` - `cortex_alertmanager_state_initial_sync_total` - `cortex_alertmanager_state_initial_sync_completed_total` - `cortex_alertmanager_state_initial_sync_duration_seconds` - `cortex_alertmanager_state_persist_total` - `cortex_alertmanager_state_persist_failed_total` Note this complements the already existing metrics: - `cortex_alertmanager_partial_state_merges_total` - `cortex_alertmanager_partial_state_merges_failed_total` - `cortex_alertmanager_state_replication_total` - `cortex_alertmanager_state_replication_failed_total` Overly detailed timing metrics have not been included, instead opting for just a single (non per-user) histogram for the duration of the initial state operation. Timings for storage read/write are not included as they are already provided from the bucket client. Signed-off-by: Steve Simpson <[email protected]>
Signed-off-by: Steve Simpson <[email protected]>
Signed-off-by: Steve Simpson <[email protected]>
Signed-off-by: Steve Simpson <[email protected]>
Signed-off-by: Steve Simpson <[email protected]>
pracucci
approved these changes
May 4, 2021
pstibrany
approved these changes
May 4, 2021
Signed-off-by: Steve Simpson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
This commit adds a number of metrics to help track the operation of the
alertmanager sharding, specifically around the handling of the state.
cortex_alertmanager_state_fetch_replica_state_totalcortex_alertmanager_state_fetch_replica_state_failed_totalcortex_alertmanager_state_initial_sync_totalcortex_alertmanager_state_initial_sync_completed_totalcortex_alertmanager_state_initial_sync_duration_secondscortex_alertmanager_state_persist_totalcortex_alertmanager_state_persist_failed_totalNote this complements the already existing metrics:
cortex_alertmanager_partial_state_merges_totalcortex_alertmanager_partial_state_merges_failed_totalcortex_alertmanager_state_replication_totalcortex_alertmanager_state_replication_failed_totalOverly detailed timing metrics have not been included, instead opting
for just a single (non per-user) histogram for the duration of the
initial state operation. Timings for storage read/write are not
included as they are already provided from the bucket client.
Checklist
Documentation addedCHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]