Skip to content

Fix flaky test_command_output_continuation test in BashSession #8813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 3, 2025

Conversation

rbren
Copy link
Collaborator

@rbren rbren commented May 30, 2025

  • This change is worth documenting at https://docs.all-hands.dev/
  • Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

End-user friendly description of the problem this fixes or functionality this introduces.
Fixes a flaky test in the BashSession class that was intermittently failing due to timing issues.


Summarize what the PR does, explaining any non-trivial design decisions.
This PR modifies the test_command_output_continuation test to be more robust against timing issues. The test was failing because it expected specific output patterns that could vary depending on execution timing.

The updated test now:

  1. Uses a more robust approach that handles both cases where the command completes immediately or times out
  2. Verifies all expected numbers (1-5) appear in the output without requiring them to appear in separate observations
  3. Removes assertions that were sensitive to timing and implementation details

This makes the test more reliable while still verifying the core functionality of command output continuation.


Link of any specific issues this addresses:
No specific issue, but this addresses a flaky test that was recently introduced.


To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:5e7a547-nikolaik   --name openhands-app-5e7a547   docker.all-hands.dev/all-hands-ai/openhands:5e7a547

@rbren rbren marked this pull request as ready for review May 30, 2025 17:42
@enyst enyst requested a review from ryanhoangt May 30, 2025 19:52
Copy link
Collaborator

@tofarr tofarr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one has been annoying me!

Copy link
Collaborator

@ryanhoangt ryanhoangt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@rbren rbren merged commit 633d5b2 into main Jun 3, 2025
19 checks passed
@rbren rbren deleted the fix-flaky-bash-session-test branch June 3, 2025 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants