Skip to content

Add important warning about not pushing/creating PRs unless explicitly asked #9357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 25, 2025

Conversation

xingyaoww
Copy link
Collaborator

@xingyaoww xingyaoww commented Jun 25, 2025

Summary

This PR adds an important warning to the <PULL_REQUESTS> section of the system prompt to address the issue where Claude-4 has been too eager to start PRs without being explicitly requested to do so by users.

Changes

  • Added **Important**: Do not push to the remote branch and/or start pull request unless explicitly asked to do so. as the first line of the <PULL_REQUESTS> section in openhands/agenthub/codeact_agent/prompts/system_prompt.j2

Context

Based on team discussion, Claude-4 has been ignoring existing prompt guidance about not opening PRs until asked, causing user confusion. This change makes the instruction more prominent with bold formatting and explicit placement at the beginning of the section.

Testing

  • Pre-commit hooks pass
  • Change is minimal and focused
  • Existing functionality preserved

@xingyaoww can click here to continue refining the PR


To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:9694cc9-nikolaik   --name openhands-app-9694cc9   docker.all-hands.dev/all-hands-ai/openhands:9694cc9

…y asked

This addresses the issue where Claude-4 has been too eager to start PRs
without being explicitly requested to do so by users.
@xingyaoww
Copy link
Collaborator Author

I'm running an eval locally for this 🙏

@xingyaoww
Copy link
Collaborator Author

This PR:

Instances submitted: 200
Instances completed: 199
Instances incomplete: 300
Instances resolved: 136
Instances unresolved: 63
Instances with empty patches: 0
Instances with errors: 1
Unstopped containers: 0
Unremoved images: 500
Report written to claude-sonnet-4-20250514_maxiter_100_N_v0.46.0-no-hint-pr9357-run_1.20250625_215337.json
MODEL_NAME_OR_PATH: claude-sonnet-4-20250514_maxiter_100_N_v0.46.0-no-hint-pr9357-run_1
RESULT_OUTPUT_DIR: /home/xingyaow/OpenHands-dev/evaluation/evaluation_outputs/outputs/princeton-nlp__SWE-bench_Verified-test/CodeActAgent/claude-sonnet-4-20250514_maxiter_100_N_v0.46.0-no-hint-pr9357-run_1
Checking for changes: 0it [00:00, ?it/s]
Updating output file: 200it [00:01, 101.96it/s]

On main (from #9140 (comment)):

Total instances: 500
Instances submitted: 200
Instances completed: 198
Instances incomplete: 300
Instances resolved: 135
Instances unresolved: 63
Instances with empty patches: 0
Instances with errors: 2
Unstopped containers: 0
Unremoved images: 500

Should be ready for review

@xingyaoww xingyaoww marked this pull request as ready for review June 25, 2025 22:47
@xingyaoww xingyaoww merged commit 743c814 into main Jun 25, 2025
19 checks passed
@xingyaoww xingyaoww deleted the add-pr-warning branch June 25, 2025 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants