Enable active series metrics in the ingester by default #4257
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change calculates and exports the `cortex_ingester_active_series` by default. Up to this point, the metric was disabled by default since calculating it consumes some amount of memory. The original PR (#3153) estimated for 1M active series at least 40MB and up to another 200MB depending on our luck reusing labels from the ref cache. We (Grafana) have been running with this setting enabled on our ingesters for some time and the resource usage doesn't appear to be significant. This feature appears to add between 1.2 - 1.6% in memory usage when enabled: ~140MB out of a total of ~10GB of memory used per ingester. The ingesters I measured this on * Have multiple tenants running production workloads * Have about 1.3M active series each * Have about a 10GB working set (as measured by `kubectl top` and exported k8s metrics) Based on this and the utility of the metric itself, I'd like to enable it by default. Screenshots of the pprof heap output attached. Signed-off-by: Nick Pillitteri <[email protected]>
Signed-off-by: Nick Pillitteri <[email protected]>
56quarters
changed the title
stevesg
approved these changes
Jun 4, 2021
Signed-off-by: Nick Pillitteri <[email protected]>
pstibrany
approved these changes
Jun 8, 2021
There was a problem hiding this comment.
LGTM, thank you.
Up to this point, the metric was disabled by default since calculating it consumes some amount of memory.
That's not the only reason. Main reason is that everybody does a calculation of active series differently.
This feature was introduced because after switch from chunks to blocks engine, "in memory series" started to be computed differently. To make it more configurable and compatible with computation in chunks storage, we have introduced "active series" metrics.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Nick Pillitteri [email protected]
What this PR does:
This change calculates and exports the
cortex_ingester_active_seriesmetric by default. Up to this point, the metric was disabled by default since
calculating it consumes some amount of memory.
The original PR (#3153) estimated for 1M active series at least 40MB
and up to another 200MB depending on our luck reusing labels from the
ref cache.
We (Grafana) have been running with this setting enabled on our ingesters
for some time and the resource usage doesn't appear to be significant.
This feature appears to add between 1.2 - 1.6% in memory usage when
enabled: ~140MB out of a total of ~10GB of memory used per ingester.
The ingesters I measured this on
kubectl topand exportedk8s metrics)
Based on this and the utility of the metric itself, I'd like to enable it
by default.
Screenshots of the pprof heap output attached.
Checklist
- [ ] Tests updatedCHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]