Skip to content

Integration test github action #5076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
enyst opened this issue Nov 16, 2024 · 2 comments · Fixed by #5077
Closed

Integration test github action #5076

enyst opened this issue Nov 16, 2024 · 2 comments · Fixed by #5077
Labels
enhancement New feature or request fix-me Attempt to fix this issue with OpenHands

Comments

@enyst
Copy link
Collaborator

enyst commented Nov 16, 2024

The eval-runner workflow from the .github/workflows directory is too big. Read it all carefully, and note how it's doing two different things: integration test evaluation and SWE-Bench evaluation. Let's split it:

  1. find the section named "Run integration test evaluation" and the necessary sections before and after
  2. create a new workflow named integration-runner
  3. define it as to work on a PR with the label 'integration-test' trigger
  4. move the integration test evaluation section to the new file and copy the prerequisites.

IMPORTANT:

  • the action should run only when labeled or manually triggered.
@enyst enyst added enhancement New feature or request fix-me Attempt to fix this issue with OpenHands labels Nov 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

OpenHands started fixing the issue! You can monitor the progress here.

@github-actions GitHub Actions
Copy link
Contributor

A potential fix has been generated and a draft PR #5077 has been created. Please review the changes.

@openhands-agent openhands-agent mentioned this issue Nov 16, 2024
Merged
enyst pushed a commit to enyst/playground that referenced this issue Nov 23, 2024
@openhands-agent
Fix issue All-Hands-AI#5076: Integration test github action
eaf3057
enyst added a commit to enyst/playground that referenced this issue Nov 25, 2024
@enyst @openhands-agent
Int/main/openhands fix issue 5076 (#10)
6ddae01 
* Fix issue All-Hands-AI#5076: Integration test github action

* Update integration-runner.yml

* Update integration-runner.yml

* update variables

* use haiku

* use base url

* fix report name

* Fix pr #8: Integration tests (openhands fix issue 5076)

* Revert "Fix pr #8: Integration tests (openhands fix issue 5076)"

This reverts commit dcd4681.

* Fix pr #8: Integration tests (openhands fix issue 5076)

* use haiku explicitly, in results too

* remove duplicate

* Update .github/workflows/integration-runner.yml

* Revert "Update .github/workflows/integration-runner.yml"

This reverts commit 7e7200e.

* funny space

* Fix pr #8: Integration tests (openhands fix issue 5076)

* artifact fix

* clean up remote runtimes

* clean up runtimes more aggressively - a bit unexpected though

* Fix pr #8: Integration tests (openhands fix issue 5076)

* fix type issue that was preventing checking results

* try with waiting time

* add eval notes

* increase timeouts

* try with CI local builds

* fix eval output

* set debug

* fix tests!

* fix outputs

* keep details in logs, not github comment

* tweak schedule

* lint-y

---------

Co-authored-by: openhands <[email protected]>
enyst added a commit that referenced this issue Nov 27, 2024
@openhands-agent @enyst
Fix issue #5076: Integration test github action (#5077)
f0ca223 
Co-authored-by: Engel Nyst <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request fix-me Attempt to fix this issue with OpenHands
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix issue #5076: Integration test github action All-Hands-AI/OpenHands
1 participant
@enyst