Prevent failed ingestion from affecting rate limiting in distributor. #3825
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stevesg
force-pushed
the
distributor-rate
branch
2 times, most recently
from
79ec0c6 to
d38b122
Compare
|
Fixed a couple of typos in comments, and added issue number to changelog. |
stevesg
force-pushed
the
distributor-rate
branch
2 times, most recently
from
35f5a4f to
76570af
Compare
|
Rebase, change issue number to PR number. |
Change the handling of the rate limiting in the distributor such that when the ingestion fails, the reservation in the rate limiter is canceled, returning the tokens for future use in the proceeding requests. Signed-off-by: Steve Simpson <[email protected]>
stevesg
force-pushed
the
distributor-rate
branch
from
76570af to
0869be7
Compare
|
Fix lint |
pracucci
reviewed
Feb 15, 2021
Signed-off-by: Steve Simpson <[email protected]>
stevesg
force-pushed
the
distributor-rate
branch
from
6cf0c29 to
29f0177
Compare
Signed-off-by: Steve Simpson <[email protected]>
pracucci
approved these changes
Feb 15, 2021
Signed-off-by: Steve Simpson <[email protected]>
Signed-off-by: Steve Simpson <[email protected]>
stevesg
force-pushed
the
distributor-rate
branch
from
65c1abd to
a42432c
Compare
pstibrany
reviewed
Feb 16, 2021
Signed-off-by: Steve Simpson <[email protected]>
|
What happens in the case that some samples succeed and some fail? |
3 tasks
pracucci
added a commit
to pracucci/cortex
that referenced
this pull request
Mar 12, 2021
…ributor. (cortexproject#3825)" This reverts commit 9ba848b. Signed-off-by: Marco Pracucci <[email protected]>
pracucci
added a commit
that referenced
this pull request
Mar 12, 2021
…ributor. (#3825)" (#3948) This reverts commit 9ba848b. Signed-off-by: Marco Pracucci <[email protected]>
pstibrany
pushed a commit
to pstibrany/cortex
that referenced
this pull request
Mar 15, 2021
…ributor. (cortexproject#3825)" (cortexproject#3948) This reverts commit 9ba848b. Signed-off-by: Marco Pracucci <[email protected]> (cherry picked from commit 6aa2a69)
pstibrany
added a commit
that referenced
this pull request
Mar 15, 2021
* Revert "Prevent failed ingestion from affecting rate limiting in distributor. (#3825)" (#3948) This reverts commit 9ba848b. Signed-off-by: Marco Pracucci <[email protected]> (cherry picked from commit 6aa2a69) * Update VERSION Signed-off-by: Peter Štibraný <[email protected]> Co-authored-by: Marco Pracucci <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Change the handling of the rate limiting in the distributor such that when the ingestion fails, the reservation in the rate limiter is canceled, returning the tokens for future use in the proceeding requests.
In the process, a change was made to the
util/limiterpackage to useReserveN. To match the semantics ofApplyN, we have to check theDelay()on the returnedrate.Reservationis zero. I also chose to only expose a limited subset of the rate.Reservation struct (to useCancelAt), following from how the existing code does not expose thex/time/ratepackage.Unit tests added to
rate_limiter_test.goanddistributor_test.goto check the cancel behaviour.Which issue(s) this PR fixes:
Fixes #3804
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]