Fix integration tests (All-Hands-AI#1764)

* refactor remind_iterations

* regenerate tests

* concatenate iteration message

* add some helpers to the tests

* regenerate tests

* add to logs

* regenerate tests

* add debug info

* fix exit_on_message

* fix regen script

* regenerate tests

* Revert "Merge branch 'rb/test-regen' of ssh://github.com/opendevin/opendevin into rb/test-regen"

This reverts commit b9cd1ac, reversing
changes made to c888285.

* remove prints

* revert files

* revert more

* revert more

* regenerate for the last time I hope

* add back remind_iter

* regenerate

* add back remind_iter

* regenerate

* fix remind_iter

* regenerate yet again

* regen

* remove comment

* regen again

Author: rbren

opendevin/core/main.py

@@ -25,7 +25,7 @@ def read_task_from_stdin() -> str:
     return sys.stdin.read()
 
 
-async def main(task_str: str = '') -> None:
+async def main(task_str: str = '', exit_on_message: bool = False) -> None:
     """
     Main coroutine to run the agent controller with task input flexibility.
     It's only used when you launch opendevin backend directly via cmdline.
@@ -84,8 +84,10 @@ async def main(task_str: str = '') -> None:
     async def on_event(event: Event):
         if isinstance(event, AgentStateChangedObservation):
             if event.agent_state == AgentState.AWAITING_USER_INPUT:
-                message = input('Request user input >> ')
-                action = MessageAction(content=message)
+                action = MessageAction(content='/exit')
+                if not exit_on_message:
+                    message = input('Request user input >> ')
+                    action = MessageAction(content=message)
                 await event_stream.add_event(action, EventSource.USER)
 
     event_stream.subscribe(EventStreamSubscriber.MAIN, on_event)

opendevin/server/agent/agent.py

@@ -134,7 +134,9 @@ async def on_event(self, event: Event):
         Args:
             event: The agent event (Observation or Action).
         """
-        if event.source == 'agent' and not isinstance(event, (NullAction, NullObservation)):
+        if event.source == 'agent' and not isinstance(
+            event, (NullAction, NullObservation)
+        ):
             await self.send(event.to_dict())
         return
 

tests/integration/conftest.py

@@ -79,7 +79,9 @@ def mock_completion(*args, test_name, **kwargs):
     for message in messages:
         message_str += message['content']
     mock_response = get_mock_response(test_name, message_str)
-    assert mock_response is not None, 'Mock response for prompt is not found'
+    assert mock_response is not None, (
+        'Mock response for prompt is not found:\n\n' + message_str
+    )
     response = completion(**kwargs, mock_response=mock_response)
     return response
 

tests/integration/mock/CodeActAgent/test_edits/prompt_002.log

@@ -220,19 +220,18 @@ Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ----------
 
-Let's start by opening the file `bad.txt` to see what's inside and identify the typos.
-
+Sure! Let's first take a look at the content of the file `bad.txt` to identify the typos that need to be fixed. Let me display the content of the file:
 <execute_bash>
-open bad.txt
+cat bad.txt
 </execute_bash>
 
 ----------
 
 OBSERVATION:
-[File: /workspace/bad.txt (4 lines total)]
-1:This is a stupid typoo.
-2:Really?
-3:No mor typos!
-4:Enjoy!
+This is a stupid typoo.
+Really?
+No mor typos!
+Enjoy!
+
 ENVIRONMENT REMINDER: You have 9 turns left to complete the task.
 [Command -1 finished with exit code 0]]

tests/integration/mock/CodeActAgent/test_edits/prompt_003.log

@@ -220,61 +220,37 @@ Fix typos in bad.txt. Do not ask me for confirmation at any point.
 
 ----------
 
-Let's start by opening the file `bad.txt` to see what's inside and identify the typos.
-
+Sure! Let's first take a look at the content of the file `bad.txt` to identify the typos that need to be fixed. Let me display the content of the file:
 <execute_bash>
-open bad.txt
+cat bad.txt
 </execute_bash>
 
 ----------
 
 OBSERVATION:
-[File: /workspace/bad.txt (4 lines total)]
-1:This is a stupid typoo.
-2:Really?
-3:No mor typos!
-4:Enjoy!
+This is a stupid typoo.
+Really?
+No mor typos!
+Enjoy!
+
 ENVIRONMENT REMINDER: You have 9 turns left to complete the task.
 [Command -1 finished with exit code 0]]
 
 ----------
 
-I see a few typos in the file `bad.txt`. Let's correct them:
-
-1. "typoo" should be "typo".
-2. "mor" should be "more".
-
-I will now edit the file to fix these typos.
-
+I see the typos that need to be fixed in the file `bad.txt`. Let's correct them without asking for confirmation. Let me open the file for editing:
 <execute_bash>
-edit 1:1
-This is a stupid typo.
-end_of_edit
-edit 3:3
-No more typos!
-end_of_edit
+open bad.txt
 </execute_bash>
 
 ----------
 
 OBSERVATION:
-> This is a stupid typo.
-> EOF
-[File: /workspace/bad.txt (4 lines total)]
-1:This is a stupid typo.
-2:Really?
-3:No mor typos!
+[File: /workspace/bad.txt (4 lines total)]
+1:This is a stupid typoo.
+2:Really?
+3:No mor typos!
 4:Enjoy!
-File updated. Please review the changes and make sure they are correct (correct indentation, no duplicate lines, etc). Edit the file again if necessary.edit 3:3 <<EOF
-> No more typos!
-> EOF
 
-
-[File: /workspace/bad.txt (4 lines total)]
-1:This is a stupid typo.
-2:Really?
-3:No more typos!
-4:Enjoy!
-File updated. Please review the changes and make sure they are correct (correct indentation, no duplicate lines, etc). Edit the file again if necessary.
 ENVIRONMENT REMINDER: You have 8 turns left to complete the task.
 [Command -1 finished with exit code 0]]