Fix crash at exit due to a race condition with new signal handler #2545
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, the signal handler would interrupt the main thread (thread that calls `Server::Run`), so there was no chance for the main thread to resume before the signal handler was finished. With the new signal handler implementation in gz::common, the signal handler is now called from a different thread which means the main thread might resume before the signal handler is finished. This causes a race condition in which the `Server` class, and therefore the `ServerPrivate` class are being destructed while the signal handler is still active. Since `ServerPrivate` also contains the class that encapsulates the signal handler implementation, the causes a crash as the signal handler tries to access data that has been deleted. The fix here is to ensure that the main thread does not exit until the signal handler is finished by synchronizing on a mutex. Signed-off-by: Addisu Z. Taddese <[email protected]>
|
CI caught some new test failures. I was able to reproduce the |
It's incorrect to decide whether we call SimulationRunner::Stop on the `running` variable since that could be set to false when after a fixed number of iterations has been executed. i.e., the running variable does not really indicate that we're in shutdown mode. Signed-off-by: Addisu Z. Taddese <[email protected]>
Apparently I was not running tests that need a display because I didn't have glxinfo available on the container I was using. I have pushed a change that fixed |
iche033
approved these changes
Aug 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🦟 Bug fix
Fixes #
Summary
Previously, the signal handler would interrupt the main thread (thread that calls
Server::Run), so there was no chance for the main thread to resume before the signal handler was finished. With the new signal handler implementation in gz::common, the signal handler is now called from a different thread which means the main thread might resume before the signal handler is finished. This causes a race condition in which theServerclass, and therefore theServerPrivateclass are being destructed while the signal handler is still active. SinceServerPrivatealso contains the class that encapsulates the signal handler implementation, the causes a crash as the signal handler tries to access data that has been deleted.The fix here is to ensure that the main thread does not exit until the signal handler is finished by synchronizing on a mutex.
Checklist
codecheckpassed (See contributing)Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining
Signed-off-bymessages.