Add new metrics to aid troubleshooting tombstone convergence. #4231
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stevesg
force-pushed
the
memberlist-metrics
branch
2 times, most recently
from
3b45a7d to
91cb47b
Compare
pracucci
approved these changes
May 31, 2021
- `memberlist_client_kv_store_value_tombstones`
Expose the number of tombstones currently in the value for each key
in the store. If the tombstones are being propagated as expected,
then each instance should have the same value for this metric.
- `memberlist_client_kv_store_value_tombstones_removed_total`
Count the number of tombstones which have been removed because they
aged out. This can be used to troubleshoot whether tombstones are
being removed too soon (i.e. before the state is converged).
- `memberlist_client_messages_to_broadcast_dropped_total`
Count occurrences of messages being quietly dropped when they were
expecting to have been broadcast. As log messages already existed
for these events, the logs have had the `key` field added instead
of exposing per-key metrics, for consistency with other metrics
of this nature in the package.
Signed-off-by: Steve Simpson <[email protected]>
Signed-off-by: Steve Simpson <[email protected]>
stevesg
force-pushed
the
memberlist-metrics
branch
from
1b0826f to
dfb80dc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Add new metrics to aid troubleshooting tombstone convergence.
memberlist_client_kv_store_value_tombstonesExpose the number of tombstones currently in the value for each key
in the store. If the tombstones are being propagated as expected,
then each instance should have the same value for this metric.
memberlist_client_kv_store_value_tombstones_removed_totalCount the number of tombstones which have been removed because they
aged out. This can be used to troubleshoot whether tombstones are
being removed too soon (i.e. before the state is converged).
memberlist_client_messages_to_broadcast_dropped_totalCount occurrences of messages being quietly dropped when they were
expecting to have been broadcast. As log messages already existed
for these events, the logs have had the
keyfield added insteadof exposing per-key metrics, for consistency with other metrics
of this nature in the package.
Checklist
Documentation addedCHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]