fix race in mock publisher #1560
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FakePublisherServiceImpl::publish has a race. If the call to publish happen before a response can be placed, the server will try to respond with a null object. The fix is to either - always set the response before calling - make publish wait for the response For propriety, this commit does both. This fix reveals another flake. Publisher uses exponential backoff with jitter. The jitter randomly picks a number between 0 and a maximum. If we pick low values too many times, it will retry too often and the server will run out of canned transient errors to respond back with. The test still passed since it expected any Throwable. This commit fixed the test to expect FakeException, set the jitter to random in range (max/2, max), and increases the number of canned errors to compensate. Retrying can still causes random test failures, independently of above changes. If a request fails due to DEADLINE_EXCEEDED, the future is completed with a corresponding error. However, the last RPC might not have been successfully cancelled. When a new test starts, it gives canned response to the server. The server might use some of these responses to respond to RPCs of previous tests. Consequently, a misbehaving test can fail every test that comes after it. This commit changes the test setup code so that it creates a new fake server for every test to avoid this problem.
googlebot
added
the
cla: yes
|
cc @davidtorres , we still seem to have flakes in |
|
Couldn't you also mock the jitter, or the random number provider? |
|
@garrettjonesgoogle Good idea. PTAL |
|
LGTM, after you update the PR description to match the latest logic. |
|
Description fixed in PR and commit |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
FakePublisherServiceImpl::publish has a race.
If the call to publish happen before a response can be placed,
the server will try to respond with a null object.
The fix is to either
For propriety, this commit does both.
This fix reveals another flake.
Publisher uses exponential backoff with jitter.
The jitter randomly picks a number between 0 and a maximum.
If we pick low values too many times,
it will retry too often and the server will run out of canned
transient errors to respond back with.
The test still passed since it expected any Throwable.
This commit fixed the test to expect FakeException,
and use a fake "random" number generator to ensure
we don't run out of return values.
Retrying can still causes random test failures,
independently of above changes.
If a request fails due to DEADLINE_EXCEEDED,
the future is completed with a corresponding error.
However, the last RPC might not have been successfully cancelled.
When a new test starts, it gives canned response to the server.
The server might use some of these responses to respond to
RPCs of previous tests.
Consequently, a misbehaving test can fail every test that comes
after it.
This commit changes the test setup code so that it
creates a new fake server for every test to avoid this problem.