Skip to content

fix: Agentskills enhancements #2384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 36 commits into from
Jun 16, 2024
Merged

Conversation

tobitege
Copy link
Collaborator

@tobitege tobitege commented Jun 11, 2024

Should fix: #1890 [Bug]: CodeAct repeatedly edits the wrong file
Followup to: #2346 feat: append_file incl. all tests

  • agentskills.py updated: append_file, edit_file, open_file, _print_window revised/refactored

  • test_agent_skill.py unit tests partially refactored and updated

  • updated CodeActAgent prompts for new signatures; added system prompt words for file handling.

  • test_agent.py integration tests retested

  • couple more revisions to docs

  • CodeActSWEAgent: only updated method signatures for agent skills

  • The open_file method got a new int parameter context_lines (default: 100) which allows the agent to get smaller or bigger text chunks of up to 2000 lines (I made the decision for that cap, agent can still use scroll methods)

  • The append_file and edit_file methods are basically stumps now to call an internal method that combines both functionalities (bool isAppend parameter).

  • Revised the _print_window method to handle "edge" cases better and keep the current line centered if possible

  • Made sure that empty files are handled better, i.e. line count is 1, not 0; we get 1-based lines back from the linter, so 1-based is preferably

@tobitege tobitege marked this pull request as draft June 11, 2024 06:29
@tobitege tobitege changed the title [DRAFT] Agentskills enhancements Agentskills enhancements - Workflow start, no merge please Jun 14, 2024
@tobitege tobitege marked this pull request as ready for review June 14, 2024 10:14
@tobitege
Copy link
Collaborator Author

cc @neubig @li-boxuan Kindly ask for a review and "Workflows" unlock, to see if test run fine on CI.

@tobitege tobitege changed the title Agentskills enhancements - Workflow start, no merge please Agentskills enhancements Jun 14, 2024
@tobitege
Copy link
Collaborator Author

tobitege commented Jun 14, 2024

Working on more fixes...

@tobitege tobitege marked this pull request as draft June 15, 2024 03:03
@tobitege
Copy link
Collaborator Author

tobitege commented Jun 15, 2024

chugging away...

@tobitege
Copy link
Collaborator Author

tobitege commented Jun 15, 2024

@li-boxuan I did update my branch to current main (cherry-picked those 4 last commits), but still see branch conflicts about deleted log files.
Those files should be gone already.

Would appreciate, if you could lend a hand on this issue. Thank you! 🤗

@li-boxuan
Copy link
Collaborator

@tobitege I cannot directly push to your branch, but I can share what I did:

git fetch upstream
git merge upstream/main
git rm -rf tests/integration/mock/CodeActSWEAgent/test_ipython_module
ONLY_TEST_AGENT=CodeActSWEAgent ./tests/integration/regenerate.sh

@tobitege
Copy link
Collaborator Author

Thanks a lot! Will try that now.

@tobitege
Copy link
Collaborator Author

@tobitege I cannot directly push to your branch, but I can share what I did:

Fantastic, that worked! Ready for review then. 🍰

@tobitege tobitege changed the title Agentskills enhancements feat: Agentskills enhancements Jun 15, 2024
@tobitege tobitege changed the title feat: Agentskills enhancements fix: Agentskills enhancements Jun 15, 2024
Copy link
Collaborator

@xingyaoww xingyaoww left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM! Thanks for doing this!! I think agentskills get a little bit more robust day to day thanks to your contribution :)

* updated docstrings with more details for LLM
* fix legacy typo in prompts causing ]] instead of ]
* several mock files regenerated
@tobitege
Copy link
Collaborator Author

tobitege commented Jun 16, 2024

@xingyaoww latest commit should address all raised issues; even found another bug I was able to fix in between.

Might be a legacy issue here (with LLM):
the LLM does not always grasp, that "edit_file" for a specific line means replacement, even though it says so in the prompt.
It leisurely then assumes it is an insert and ends up replacing the given line.
That just happened to me earlier with gpt4o and I checked the made changes with BeyondCompare (don't ask why lol).
I again slightly changed the prompt to make it aware of that, to be safe.

@tobitege tobitege requested a review from xingyaoww June 16, 2024 08:37
@tobitege tobitege requested review from yufansong and rbren June 16, 2024 15:19
@tobitege
Copy link
Collaborator Author

Hmm seems I missed to regen an integration test. Will fix asap.

Copy link
Contributor

@neubig neubig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for this, it seems this will make things much more solid! I have left a bunch of comments, please re-request review when you have handled them.

Auxiliary comment not related to this PR in particular, but agentskills more in general: I was confused by print(). It seems that it's clearly not just for writing to the terminal, but rather it's functioning more like return. But this isn't documented anywhere in the python file comments. It'd be good to add a comment about this at the top of the file. Also, maybe we should consider using return instead of print to make this more like canonical Python? It doesn't need to be fixed in this PR, but maybe we could open an issue about it.

Also, one request for the future -- this PR is rather large which caused reviewing it to take a significant amount of time. If possible, you can also split into somewhat smaller PRs to reduce the reviewing burden somewhat and allow us to have faster turnaround on reviews. Thanks a lot!

@tobitege
Copy link
Collaborator Author

Thanks so much for taking the time reviewing this (grown) PR!
A lot of the regenerated integration files "bloated" this more than expected.
I'll definitely take it to heart to make smaller PR's with this topic next time. 😁

@tobitege tobitege requested a review from neubig June 16, 2024 18:16
Copy link
Contributor

@neubig neubig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks!

@neubig neubig merged commit 823298e into All-Hands-AI:main Jun 16, 2024
2 checks passed
@tobitege tobitege deleted the agent-fileops branch June 16, 2024 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: CodeAct repeatedly edits the wrong file
6 participants