Skip to content

Not Marking UI Elements #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
southpawmurph opened this issue May 8, 2025 · 0 comments
Open

Not Marking UI Elements #2

southpawmurph opened this issue May 8, 2025 · 0 comments

Comments

@southpawmurph
Copy link

southpawmurph commented May 8, 2025

Overview: The agent isn't marking the UI elements that are in the area of interest.

Details: The agent is being tasked with finding the Education section of the job application and selecting a value in the degree dropdown. The agent scrolls through the page but it visually isn't marking the UI elements in the section of interest. The only areas it marks the UI elements is at the header and footer of the web page. This makes me wonder if the agent isn't able to see the UI elements between the header and footer.
What other information can I provide to help identify the root cause for this issue I'm facing.

Code:
from langchain_google_genai import ChatGoogleGenerativeAI
from browser_use import Agent, Browser
from dotenv import load_dotenv
load_dotenv()

llm = ChatGoogleGenerativeAI(model='gemini-2.0-flash', api_key='')

import asyncio
async def main():
browser = Browser()
agent = Agent(
task="""Start from the link.
Link: https://careers-unitypoint.icims.com/jobs/164669/data-governance-analyst-ii/candidate?mode=prepopulate&mobile=false&width=1230&height=500&bga=true&needsRedirect=false&jan1offset=-360&jun1offset=-300
Select "Bachelor's" degree in the Education section.""",
llm=llm,
browser=browser,
use_vision=True
)
await agent.run()
await browser.close()
if name == "main":
asyncio.run(main())

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant