Not all HTML elements are being detected by browser-use #1287

trevorstr · 2025-04-04T20:22:44Z

trevorstr
Apr 4, 2025

I'm trying to automate login to a website, and browser-use is not detecting all of the elements on the page. It's only detecting some of the elements at the top.

Has anyone else seen this behavior? Is there any way to force it to "see" all the elements on the page, particularly the text input elements to provide login credentials?

Why is it only detecting a couple of elements up at the top?

    agent = Agent(
        task=' '.join([
            "Go to the AWS Partner Portal: http://partnercentral.awspartner.com/partnercentral2/.", 
            "Make sure you detect all of the user text input elements on the page."
            "Login with business email 'test@test.com' and password 'asdfasdf'",
        ]),
        llm=ChatGoogleGenerativeAI(
            model='gemini-2.0-flash-exp',
            api_key=api_key,
        ),
        # controller=controller,
    )
    await agent.run()

pirate · 2025-04-04T22:16:12Z

pirate
Apr 4, 2025

This page uses a cross-origin iframe which means you have to set disable_security=True for browser-use to be able to work.

...
browser = Browser(
    config=BrowserConfig(
+        disable_security=True,
        new_context_config=BrowserContextConfig(
+	        disable_security=True,
        ),
    ),
)

Make sure not to visit any untrusted sites when you disable security, as they could use your cookies or login info for malicious purposes.

6 replies

trevorstr Apr 4, 2025
Author

Unfortunately that did not work.

FYI: I changed to using Bedrock instead of Gemini, although I don't think that should matter.

Here's the full code.

Included the logs. Even though it can "see" the form input fields now, it doesn't seem to be able to insert any input into them. 🤔

import asyncio
import boto3
from langchain_aws import ChatBedrockConverse
from browser_use import Agent, Browser, BrowserConfig, BrowserContextConfig

async def main():
    bedrock_client = boto3.client('bedrock-runtime', region_name='us-west-2')
    llm = ChatBedrockConverse(
        model_id='us.anthropic.claude-3-5-sonnet-20241022-v2:0',
        temperature=0.0,
		max_tokens=None,
		client=bedrock_client,
    )

    browser = Browser(config=BrowserConfig(
        disable_security=True, 
        new_context_config=BrowserContextConfig(
            disable_security=True
        )
    ))

    agent = Agent(
        browser=browser,
        llm=llm,
        task=' '.join([
            "Go to AWS Partner Central: http://partnercentral.awspartner.com/partnercentral2/",
            "Login with email 'test@test.com' and password 'blah'",

        ])
    )
    await agent.run()

asyncio.run(main())

The output has changed, and the logs indicate the fields are being detected now, but I only see the bounding boxes drawn around the items in the original screenshot. And it never actually fills out any information in the fields.

INFO     [browser_use] BrowserUse logging setup complete with level info
INFO     [root] Anonymized telemetry enabled. See https://docs.browser-use.com/development/telemetry for more information.
/Users/trevor.sullivan/browser-use/.venv/lib/python3.13/site-packages/browser_use/agent/message_manager/views.py:59: LangChainBetaWarning: The function `load` is in beta. It is actively being worked on, so the API may change.
  value['message'] = load(value['message'])
INFO     [agent] 🚀 Starting task: Go to AWS Partner Central: http://partnercentral.awspartner.com/partnercentral2/ Login with email 'test@test.com' and password 'blah'
INFO     [agent] 📍 Step 1
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [botocore.tokens] Loading cached SSO token for stratusgrid
INFO     [agent] 🤷 Eval: Unknown - Starting fresh with an empty page
INFO     [agent] 🧠 Memory: Starting the task to login to AWS Partner Central. Need to navigate to http://partnercentral.awspartner.com/partnercentral2/ and login with email: test@test.com and password: blah
INFO     [agent] 🎯 Next goal: Navigate to AWS Partner Central login page
INFO     [agent] 🛠️  Action 1/1: {"go_to_url":{"url":"http://partnercentral.awspartner.com/partnercentral2/"}}
INFO     [controller] 🔗  Navigated to http://partnercentral.awspartner.com/partnercentral2/
INFO     [agent] 📍 Step 2
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [agent] 👍 Eval: Success - Successfully navigated to AWS Partner Central login page
INFO     [agent] 🧠 Memory: On AWS Partner Central login page. Need to login with email: test@test.com and password: blah. The login form is partially visible but need to scroll to see it completely.
INFO     [agent] 🎯 Next goal: Scroll down to see the login form completely
INFO     [agent] 🛠️  Action 1/1: {"scroll_down":{}}
INFO     [controller] 🔍  Scrolled down the page by one page
INFO     [agent] 📍 Step 3
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [agent] 👍 Eval: Success - Page has been scrolled and login form is visible
INFO     [agent] 🧠 Memory: On AWS Partner Central login page. Need to login with email: test@test.com and password: blah. Looking for login form input fields.
INFO     [agent] 🎯 Next goal: Extract content to find login form input fields
INFO     [agent] 🛠️  Action 1/1: {"extract_content":{"goal":"Find all input fields for login form including their indexes"}}
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [controller] 📄  Extracted from page
: {
  "login_form": {
    "input_fields": [
      {
        "index": 1,
        "type": "email",
        "label": "Business email",
        "placeholder": "Enter your business email",
        "validation_message": "Enter a valid Business Email"
      },
      {
        "index": 2,
        "type": "password",
        "label": "Password",
        "placeholder": "Enter your password",
        "associated_link": "Forgot password"
      }
    ],
    "submit_button": {
      "text": "Sign in"
    }
  }
}

INFO     [agent] 📍 Step 4
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [agent] 👍 Eval: Success - Found login form input fields with indexes 1 and 2
INFO     [agent] 🧠 Memory: On AWS Partner Central login page. Login form is visible. Need to fill email: test@test.com and password: blah. Will fill both fields and then look for sign in button.
INFO     [agent] 🎯 Next goal: Fill in the login form with provided credentials
INFO     [agent] 🛠️  Action 1/2: {"input_text":{"index":1,"text":"test@test.com"}}
INFO     [agent] 🛠️  Action 2/2: {"input_text":{"index":2,"text":"blah"}}
ERROR    [agent] ❌ Result failed 1/3 times:
 Error executing action input_text: Failed to input text into index 1
INFO     [agent] 📍 Step 5
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [agent] ⚠ Eval: Failed - Previous attempt to input text failed, need to find the correct input fields
INFO     [agent] 🧠 Memory: On AWS Partner Central login page. Need to login with email: test@test.com and password: blah. Previous attempt to input text failed, need to find the correct input fields first.
INFO     [agent] 🎯 Next goal: Extract content to find all input fields and buttons on the login form
INFO     [agent] 🛠️  Action 1/1: {"extract_content":{"goal":"Find all input fields and buttons on the login form including their indexes"}}
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [controller] 📄  Extracted from page
: {
  "form_elements": {
    "input_fields": [
      {
        "index": 1,
        "type": "text/email",
        "label": "Business email",
        "placeholder": "Enter your business email"
      },
      {
        "index": 2,
        "type": "password",
        "label": "Password",
        "placeholder": "Enter your password"
      }
    ],
    "buttons": [
      {
        "index": 1,
        "type": "submit",
        "text": "Sign in"
      }
    ],
    "links": [
      {
        "index": 1,
        "text": "Forgot password"
      },
      {
        "index": 2,
        "text": "Join AWS Partner Network"
      }
    ]
  }
}

INFO     [agent] 📍 Step 6
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [agent] 👍 Eval: Success - We are on the correct login page with the form
INFO     [agent] 🧠 Memory: On AWS Partner Central login page. Need to login with email: test@test.com and password: blah. Login form is partially visible but need to scroll to see input fields.
INFO     [agent] 🎯 Next goal: Scroll down to see the full login form
INFO     [agent] 🛠️  Action 1/1: {"scroll_down":{}}
INFO     [controller] 🔍  Scrolled down the page by one page
INFO     [agent] 📍 Step 7
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [agent] 👍 Eval: Success - We are on the login page but need to see the input fields
INFO     [agent] 🧠 Memory: On AWS Partner Central login page. Need to login with email: test@test.com and password: blah. Currently at bottom of page, need to scroll up to see login form.
INFO     [agent] 🎯 Next goal: Scroll up to see the login form input fields
INFO     [agent] 🛠️  Action 1/1: {"scroll_up":{}}
INFO     [controller] 🔍  Scrolled up the page by one page
INFO     [agent] 📍 Step 8
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [agent] 👍 Eval: Success - We are now on the login page with the form visible
INFO     [agent] 🧠 Memory: On AWS Partner Central login page. Need to login with email: test@test.com and password: blah. Login form is visible but need to find the correct input field indexes.
INFO     [agent] 🎯 Next goal: Extract content to find login form input fields
INFO     [agent] 🛠️  Action 1/1: {"extract_content":{"goal":"Find all input fields and buttons on the login form including their indexes"}}
INFO     [langchain_aws.chat_models.bedrock_converse] Using Bedrock Converse API to generate response
INFO     [controller] 📄  Extracted from page
: {
  "form_elements": {
    "input_fields": [
      {
        "index": 1,
        "type": "text/email",
        "label": "Business email",
        "placeholder": "Enter your business email"
      },
      {
        "index": 2,
        "type": "password",
        "label": "Password",
        "placeholder": "Enter your password"
      }
    ],
    "buttons": [
      {
        "index": 1,
        "type": "submit",
        "text": "Sign in"
      }
    ],
    "links": [
      {
        "index": 1,
        "text": "Forgot password"
      },
      {
        "index": 2,
        "text": "Join AWS Partner Network"
      }
    ]
  }
}

pirate Apr 4, 2025

Ah my mistake the page actually uses shadow roots and not iframes. We also struggle sometimes with certain pages that use shadow-DOMs extensively. #857 should improve this somewhat, for now you should remove that disable_security=True as it's not needed.

Can you try telling the agent in your prompt to click the element with the CSS selector #email-28 and then type in the email? If it works using the selector then we know it's an element detection bug, if it still doesn't work then you likely have to wait for #857 to get merged.

trevorstr Apr 4, 2025
Author

Unfortunately it's still a no-go. Here's a video of the repro.

2025-04-04.16-44-53.mp4

import asyncio
import boto3
from langchain_aws import ChatBedrockConverse
from browser_use import Agent, Browser, BrowserConfig, BrowserContextConfig

async def main():
    bedrock_client = boto3.client('bedrock-runtime', region_name='us-west-2')
    llm = ChatBedrockConverse(
        model_id='us.anthropic.claude-3-5-sonnet-20241022-v2:0',
        temperature=0.0,
		max_tokens=None,
		client=bedrock_client,
    )
    agent = Agent(
        llm=llm,
        task=' '.join([
            "Go to AWS Partner Central: http://partnercentral.awspartner.com/partnercentral2/",
            "Click the text input element with CSS selector '#email-28' and type 'test@test.com'."
            # "Login with email 'test@test.com' and password 'blah'",

        ])
    )
    await agent.run()

asyncio.run(main())

trevorstr Apr 4, 2025
Author

Looks like PR #857 is just about ready. I'll definitely update browser-use once it's merged and try it out then.

pirate Apr 4, 2025

cc @gregpr07 test this site with your new element detection fixes when you get a chance

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not all HTML elements are being detected by browser-use #1287

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Not all HTML elements are being detected by browser-use #1287

Uh oh!

trevorstr Apr 4, 2025

Replies: 1 comment · 6 replies

Uh oh!

Uh oh!

pirate Apr 4, 2025

Uh oh!

trevorstr Apr 4, 2025 Author

Uh oh!

pirate Apr 4, 2025

Uh oh!

trevorstr Apr 4, 2025 Author

Uh oh!

trevorstr Apr 4, 2025 Author

Uh oh!

pirate Apr 4, 2025

trevorstr
Apr 4, 2025

Replies: 1 comment 6 replies

pirate
Apr 4, 2025

trevorstr Apr 4, 2025
Author

trevorstr Apr 4, 2025
Author

trevorstr Apr 4, 2025
Author