[Eval,Arch] Update GPTQ eval and add `headless_mode` for Controller #2994

xingyaoww · 2024-07-18T13:32:59Z

What is the problem that this fixes or functionality that this introduces? Does it fix any open issues?

Give a summary of what the PR does, explaining any non-trivial design decisions

Add thought argument to AgentFinishAction for CodeAct.
Update GPQA inference code to be more aligned with the actual experiment we run in the paper by applying changes from this reproducible patch https://huggingface.co/spaces/OpenDevin/evaluation/blob/main/outputs/gpqa/reproducibility.patch
Add a headless_mode for controller (mainly for evaluation, default to False).
- When headless_mode is True, we will stop the whole run when the delegated agent (e.g., BrowsingAgent in the CodeAct -> Browsing delegation) hits the max iteration limit. Currently, the behavior is the delegated agent will wait for the user to hit the continue button, which puts an evaluation into an infinite loop since it is running headless.
- I can apply changes to all evaluation's run_infer.py if people like this idea!

Other references

tobitege · 2024-07-18T13:40:01Z

@xingyaoww this made my day: 🤣
"Again you are being told a million times to first report the answer in the requested format"

xingyaoww · 2024-07-18T13:42:07Z

@tobitege This was original written by @1jsingh in that reproducibility patch 😆

neubig

Thanks, looks good to me and I agree that headless mode is a good idea. I had a few comments though.

evaluation/gpqa/README.md

evaluation/gpqa/run_infer.py

neubig · 2024-07-19T02:25:35Z

evaluation/gpqa/run_infer.py

+MOST IMPORTANT: Format your response as follows:
+<<FINAL_ANSWER||
+<insert correct answer here, must be one of A, B, C, D> (Please dont use any additional characters. Just the letter of the correct answer (A/B/C/D).)
+||FINAL_ANSWER>>

-        Additional Instructions:
-        - You should ONLY interact with the environment provided to you AND NEVER ASK FOR HUMAN HELP.
-        """
+Additional Instructions:
+- Do not try to solve the question in a single step. Break it down into smaller steps.
+- You should ONLY interact with the environment provided to you AND NEVER ASK FOR HUMAN HELP.

-        # NOTE: You can actually set slightly different instruction for different agents
-        instruction += AGENT_CLS_TO_INST_SUFFIX[agent.__class__.__name__]
+- SUPER IMPORTANT: When you have reported the answer to the user in the requested format, (and only once that is done) in the next turn, please run the following command: <execute_bash> exit </execute_bash>.
+- Again you are being told a million times to first report the answer in the requested format (see again below for reference) before exiting. DO NOT EXIT WITHOUT REPORTING THE ANSWER FIRST.
+    That is, when you have decided on the answer report in the following format:
+
+    <<FINAL_ANSWER||
+    <insert correct answer here, must be one of A, B, C, D> (Please dont use any additional characters. Just the letter of the correct answer (A/B/C/D).)
+    ||FINAL_ANSWER>>
+    <execute_bash> exit </execute_bash>


This is duplicated with the codeact user response above, could we deduplicate?

Because these were the code we used to run experiments in our paper, maybe we should leave them as is for reproducibility and improve in future iterations?

We could, but because the strings are the same, wouldn't deduplicating be functionally equivalent?

Co-authored-by: Graham Neubig <[email protected]>

li-boxuan · 2024-07-19T03:04:45Z

opendevin/controller/agent_controller.py

+                f'[Agent Controller {self.id}] Delegate state: {delegate_state}'
+            )
+            if delegate_state == AgentState.ERROR or (
+                self.headless_mode and delegate_state == AgentState.PAUSED


I suggest we change await self.set_agent_state_to(AgentState.PAUSED) code to await self.set_agent_state_to(AgentState.ERROR) in headless mode here

Good idea! pushed the changes in the latest commits.

xingyaoww · 2024-07-19T03:23:52Z

opendevin/core/main.py

@@ -40,6 +40,7 @@ async def run_agent_controller(
    sandbox: Sandbox | None = None,
    runtime_tools_config: dict | None = None,
    sid: str | None = None,
+    headless_mode: bool = False,


Would it be a good idea to default this to True, so that we only need to change this to False in the backend for the application without touching all over the places (e.g., tests, evaluations)?

Yes that makes sense. When running OpenDevin via CLI, there's no way to "continue" once the application is paused.

Also agree that this is probably OK.

done in the latest commit to reduce the amount of changes in this PR

xingyaoww · 2024-07-19T03:27:47Z

evaluation/gpqa/run_infer.py

+<<FINAL_ANSWER||
+<insert correct answer here, must be one of A, B, C, D> (Please dont use any additional characters. Just the letter of the correct answer (A/B/C/D).)
+||FINAL_ANSWER>>
+""".strip()


@neubig is this similar to what you have in mind for deduplication?

AFAIK these parts are also duplicated?

'Again you are being told a million times to first report the answer in the requested format (see again below for reference) before exiting. DO NOT EXIT WITHOUT REPORTING THE ANSWER FIRST.\n' 'That is, when you have decided on the answer report in the following format:\n'

But this is not super-important, please don't let me block you.

…entSession

xingyaoww added 2 commits July 17, 2024 20:02

update and polish gptq eval

1953579

fix typo

3706ef9

xingyaoww requested a review from li-boxuan July 18, 2024 13:33

Merge branch 'main' into xw/gpqa

0d62fba

xingyaoww requested a review from neubig July 18, 2024 15:00

neubig reviewed Jul 19, 2024

View reviewed changes

xingyaoww and others added 2 commits July 19, 2024 10:53

Update evaluation/gpqa/README.md

5ab4904

Co-authored-by: Graham Neubig <[email protected]>

Update evaluation/gpqa/run_infer.py

272120f

Co-authored-by: Graham Neubig <[email protected]>

li-boxuan reviewed Jul 19, 2024

View reviewed changes

xingyaoww added 2 commits July 18, 2024 22:19

add headless mode to all appropriate agent controller call

ed52fe7

delegate set to error when in headless mode

cd5fb63

xingyaoww commented Jul 19, 2024

View reviewed changes

try to deduplicate a bit

5261489

xingyaoww commented Jul 19, 2024

View reviewed changes

neubig approved these changes Jul 19, 2024

View reviewed changes

xingyaoww added 2 commits July 19, 2024 10:15

make headless_mode default to True and only change it to false for Ag…

1f79140

…entSession

Merge branch 'main' into xw/gpqa

fd3ae43

xingyaoww enabled auto-merge (squash) July 19, 2024 21:53

fix linter

ef12cd8

xingyaoww merged commit 6b16a5d into main Jul 20, 2024

xingyaoww deleted the xw/gpqa branch July 20, 2024 03:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Eval,Arch] Update GPTQ eval and add `headless_mode` for Controller #2994

[Eval,Arch] Update GPTQ eval and add `headless_mode` for Controller #2994

xingyaoww commented Jul 18, 2024

tobitege commented Jul 18, 2024 •

edited

Loading

xingyaoww commented Jul 18, 2024

neubig left a comment

neubig Jul 19, 2024

xingyaoww Jul 19, 2024

neubig Jul 19, 2024 •

edited

Loading

li-boxuan Jul 19, 2024

xingyaoww Jul 19, 2024

xingyaoww Jul 19, 2024 •

edited

Loading

li-boxuan Jul 19, 2024

neubig Jul 19, 2024

xingyaoww Jul 19, 2024

xingyaoww Jul 19, 2024

neubig Jul 19, 2024

[Eval,Arch] Update GPTQ eval and add headless_mode for Controller #2994

[Eval,Arch] Update GPTQ eval and add headless_mode for Controller #2994

Conversation

xingyaoww commented Jul 18, 2024

tobitege commented Jul 18, 2024 • edited Loading

xingyaoww commented Jul 18, 2024

neubig left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neubig Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xingyaoww Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[Eval,Arch] Update GPTQ eval and add `headless_mode` for Controller #2994

[Eval,Arch] Update GPTQ eval and add `headless_mode` for Controller #2994

tobitege commented Jul 18, 2024 •

edited

Loading

neubig Jul 19, 2024 •

edited

Loading

xingyaoww Jul 19, 2024 •

edited

Loading