(Hotfix): Track reason for Error AgentState

[malhotra5]

• This change is worth documenting at https://docs.all-hands.dev/
• Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

End-user friendly description of the problem this fixes or functionality that this introduces.

• Track the last agent error that occurred and send it in the agent summary if the resolver fails

---

Give a summary of what the PR does, explaining any non-trivial design decisions.

This PR adds a new AgentState BUDGET_EXCEEDED when the user has ran out of LLM credits.

While we're due to simplifying the agent states, this is a simpler and fast option. This will also enable us to inform the user that they have ran out of budget when running cloud resolver.

---

Link of any specific issues this addresses.
Closes All-Hands-AI/openhands-cloud#16

---

To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:11dc511-nikolaik   --name openhands-app-11dc511   docker.all-hands.dev/all-hands-ai/openhands:11dc511

[malhotra5]

@OpenHands please take a look at the agent states, we want to make it so that AgentState.ERROR also has a field reason that accompanies it.

Maybe start by just adding a reason field to AgentStateChangedObservation? Also only populate this field when AgentState.ERROR is being assigned in the agent_controller?

[openhands-ai]

I'm on it! @malhotra5 can track my progress at all-hands.dev

[openhands-ai]

I've successfully implemented the requested changes to add a reason field to AgentState.ERROR. Here's a summary of what I did:

1. Added a reason field to the AgentStateChangedObservation class with a default empty string value.

2. Modified the set_agent_state_to method in the AgentController class to:

   • Check if the new state is AgentState.ERROR
   • Include the error reason when creating the AgentStateChangedObservation object

3. Updated the _react_to_exception method to:

   • Store the error reason in a new _last_error_reason attribute before setting the agent state
   • Use this stored reason when reporting errors
   • Set the agent state to ERROR after storing the reason

These changes ensure that when an agent enters the ERROR state, the reason for the error is captured and included in the state change observation. This will allow the UI or other components to display more detailed error information to users.

The changes have been committed and pushed to the budget-agent-state branch, updating PR #7584.

[malhotra5]

@OpenHands please fix the failing lint issue in the backend

[openhands-ai]

I'm on it! @malhotra5 can track my progress at all-hands.dev

[openhands-ai]

I fixed the failing lint issues in the backend code for PR #7584 on the budget-agent-state branch. Here's a summary of what I did:

1. Identified the lint issues by running the pre-commit checks, which revealed:

   • Trailing whitespace issues in frontend files
   • Missing newlines at end of files
   • Import ordering issues in the test_agent_controller.py file
   • Line formatting issues in the test_agent_controller.py file
   • A missing trailing comma in openhands/events/observation/agent.py

2. Fixed all the issues:

   • Properly ordered imports in test_agent_controller.py
   • Fixed line formatting in test_agent_controller.py to comply with ruff's formatting rules
   • Added the missing trailing comma in the RecallObservation.str method
   • Fixed the end-of-file newlines in the affected files

3. Committed the changes with the message "Fix lint issues in backend files"

4. Successfully pushed the changes to the budget-agent-state branch, which will update PR (Hotfix): Track reason for Error AgentState #7584

All pre-commit checks now pass for the modified files, which should resolve the failing lint issue mentioned in the PR comment.

[raymyers]

Is reason displayed, and if so will it need translations at some point?

[enyst]

I love the solution overall, we just may want to check if this works with evals, because they use state.last_error with some particular expectations for some errors:
https://github.com/search?q=repo%3AAll-Hands-AI%2FOpenHands%20last_error&type=code

At a look, I'd say it's fine - there's a function that cares about the exception name, and it seems we're setting it here anyway, and we're changing it only for a handful below?

[malhotra5]

Hmm good point, it looks like we're just saving the last error msg for eval outputs

[malhotra5]

there's a function that cares about the exception name

I can't seem to find this; I only see the following

1. Callback functions that want to set the last_error attrib to agent state
2. Evals + resolver that want to store last_error in their outputs

In this case I think this PR should be good to go

[enyst]

Ah, and this one:

OpenHands/evaluation/utils/shared.py

Line 523 in 648c8ff

def is_fatal_evaluation_error(error: str | None) -> bool:

Seems okay, the PR doesn't change those?

Edited to add: it comes from here, using last_error:

OpenHands/evaluation/benchmarks/swe_bench/run_infer.py

Line 539 in 648c8ff

if is_fatal_evaluation_error(state.last_error):

[malhotra5]

Ah thanks a bunch for linking this, it gives me peace of mind!

Yes this PR doesn't change those error messages

[enyst]

You actually make a very good point, this is not straightforward to find, and since we start using last_error, we will modify it sooner or later. Maybe a comment on the eval usage would help in the future?