[agent, browsing] Support viewing pdf and png/jpg via browser #7457
Conversation
xingyaoww
force-pushed
the
xw/view-pdf-via-browser
branch
from
cd060ee to
273437f
Compare
| @@ -18,6 +18,11 @@ | |||
| fill('a12', 'example with "quotes"') | |||
| click('a51') | |||
| click('48', button='middle', modifiers=['Shift']) | |||
|
|
|||
| You can also use the browser to view pdf, png, jpg files. | |||
| You should first check the content of /tmp/oh-server-url to get the server url, and then use it to view the file by `goto("{server_url}/view?path={absolute_file_path}")`. | |||
There was a problem hiding this comment.
I have a stupid question: isn't server_url always "127.0.0.1"?
There was a problem hiding this comment.
yes, but the port could be very different across different environment (saas, docker), and we need to tell the agent the port 😢
There was a problem hiding this comment.
Ah yeah I forgot about that.
So in the long-term, do we want to make prompts templating languages, and then just fill in the port info during runtime?
| ] | ||
|
|
||
| # Check if the file extension is supported | ||
| if file_extension not in supported_extensions: |
There was a problem hiding this comment.
@adityasoni9998 I remember you found out that files downloaded by browsergym always miss extensions in their filenames, right?
Maybe we should handle that case here as well and try to guess the file type?
There was a problem hiding this comment.
Anyways this is non-blocking
There was a problem hiding this comment.
Yeah, i think we can do that on the browsergym side of work
Co-authored-by: Boxuan Li <[email protected]>
End-user friendly description of the problem this fixes or functionality that this introduces.
Turn on vision-based browsing by default and allow OpenHands to see PDF / images in browser.
Give a summary of what the PR does, explaining any non-trivial design decisions.
Pre-req: we need to get #7452 in first
viewerin action execution server so the agent can use it to see PDF/Images directly - playwright disallowfile:///workspace/some.pdfsince it will trigger download workflow, rather than display them in browser, so we need this hack to display pdf & image.The agent can now browse through PDF in the browser:
The agent can also now see png directly in the browser:
Link of any specific issues this addresses.
To run this PR locally, use the following command: