Optimize ingester Push() on blocks storage when the per-user or per-metric series limit is reached #3971
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
pstibrany
approved these changes
Mar 18, 2021
pkg/ingester/ingester_v2.go
Outdated
| // Caused by limits. | ||
| if firstPartialErr == nil { | ||
| firstPartialErr = ve | ||
| if cause == errMaxSeriesPerUserLimitExceeded || cause == errMaxSeriesPerMetricLimitExceeded { |
There was a problem hiding this comment.
We can simplify the logic on lines 787 – 820 with:
updateFirstPartial := func(errFn func() error) {
if firstPartialErr == nil {
firstPartialErr = errFn()
}
}
switch cause {
case storage.ErrOutOfBounds:
sampleOutOfBoundsCount++
updateFirstPartial(func() error { return wrappedTSDBIngestErr(err, model.Time(s.TimestampMs), ts.Labels) })
continue
case storage.ErrOutOfOrderSample:
sampleOutOfOrderCount++
updateFirstPartial(func() error { return wrappedTSDBIngestErr(err, model.Time(s.TimestampMs), ts.Labels) })
continue
case storage.ErrDuplicateSampleForTimestamp:
newValueForTimestampCount++
updateFirstPartial(func() error { return wrappedTSDBIngestErr(err, model.Time(s.TimestampMs), ts.Labels) })
continue
case errMaxSeriesPerUserLimitExceeded:
perUserSeriesLimitCount++
updateFirstPartial(func() error { return makeLimitError(perUserSeriesLimit, i.limiter.FormatError(userID, cause)) })
continue
case errMaxSeriesPerMetricLimitExceeded:
perMetricSeriesLimitCount++
updateFirstPartial(func() error { return makeMetricLimitError(perMetricSeriesLimit, copiedLabels, i.limiter.FormatError(userID, cause)) })
continue
}
pracucci
commented
Mar 18, 2021
| } | ||
|
|
||
| var ve *validationError | ||
| if errors.As(cause, &ve) { |
There was a problem hiding this comment.
Removed because, following the code flow, I think the only errors that could be returned are per-user and per-metric series limit.
Signed-off-by: Marco Pracucci <[email protected]>
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Following up the PR #3969, in this PR I'm optimizing the
v2Push()when an high volume of per-user/series limit is reached.In the profile I've seen much CPU and memory is consumed formatting the
Limitererror. The idea in this PR is to return static errors fromLimiter, offer aFormatError()function and then format only the 1st partial error inv2Push().Metadata and chunks storage hasn't been changed.
Benchmark:
The benchmarks are on top of #3969 (I merged #3969 first and then worked on this PR):
Which issue(s) this PR fixes:
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]