Read alertmanager state from storage if peer settling fails. #4021
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Steve Simpson <[email protected]>
3 tasks
pracucci
approved these changes
Mar 31, 2021
| } | ||
| } | ||
|
|
||
| level.Info(s.logger).Log("msg", "failed to read state from storage; continuing anyway", "err", err) |
There was a problem hiding this comment.
Assuming we circuit break in case of "object not found" error, then I would raise this to warning:
Suggested change
| level.Info(s.logger).Log("msg", "failed to read state from storage; continuing anyway", "err", err) | |
| level.Warn(s.logger).Log("msg", "failed to read state from storage; continuing anyway", "err", err) |
There was a problem hiding this comment.
By circuit break, you mean don't retry any calls to object storage when no object?
There was a problem hiding this comment.
By circuit break I was meaning a "guard" (if condition then return). There's no retry mechanism here yet (to be discussed if we want it, given retries may also be offered by the bucket client).
Signed-off-by: Steve Simpson <[email protected]>
pracucci
reviewed
Apr 6, 2021
Signed-off-by: Steve Simpson <[email protected]>
2 tasks
56quarters
added a commit
to grafana/mimir
that referenced
this pull request
Jun 30, 2022
In cortexproject/cortex#3925 the ability to restore alertmanager state from peer alertmanagers was added, short-circuiting if there is only a single replica of the alertmanager. In cortexproject/cortex#4021 a fallback to read state from storage was added in case reading from peers failed. However, the short-circuiting if there is only a single peer was not removed. This has the effect of never restoring state in an alertmanager if only running a single replica. Fixes #2245 Signed-off-by: Nick Pillitteri <[email protected]>
56quarters
added a commit
to grafana/mimir
that referenced
this pull request
Jul 5, 2022
In cortexproject/cortex#3925 the ability to restore alertmanager state from peer alertmanagers was added, short-circuiting if there is only a single replica of the alertmanager. In cortexproject/cortex#4021 a fallback to read state from storage was added in case reading from peers failed. However, the short-circuiting if there is only a single peer was not removed. This has the effect of never restoring state in an alertmanager if only running a single replica. Fixes #2245 Signed-off-by: Nick Pillitteri <[email protected]>
pracucci
pushed a commit
to grafana/mimir
that referenced
this pull request
Jul 6, 2022
* Restore alertmanager state from storage as fallback In cortexproject/cortex#3925 the ability to restore alertmanager state from peer alertmanagers was added, short-circuiting if there is only a single replica of the alertmanager. In cortexproject/cortex#4021 a fallback to read state from storage was added in case reading from peers failed. However, the short-circuiting if there is only a single peer was not removed. This has the effect of never restoring state in an alertmanager if only running a single replica. Fixes #2245 Signed-off-by: Nick Pillitteri <[email protected]> * Code review changes Signed-off-by: Nick Pillitteri <[email protected]>
masonmei
pushed a commit
to udmire/mimir
that referenced
this pull request
Jul 11, 2022
* Restore alertmanager state from storage as fallback In cortexproject/cortex#3925 the ability to restore alertmanager state from peer alertmanagers was added, short-circuiting if there is only a single replica of the alertmanager. In cortexproject/cortex#4021 a fallback to read state from storage was added in case reading from peers failed. However, the short-circuiting if there is only a single peer was not removed. This has the effect of never restoring state in an alertmanager if only running a single replica. Fixes grafana#2245 Signed-off-by: Nick Pillitteri <[email protected]> * Code review changes Signed-off-by: Nick Pillitteri <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Reads the
alertmanagerstate (silences, notification log) from storage if obtaining state from a running peer fails.