fix: improve icon-only button detection using tooltips and aria-labels#4977
Open
Oxygen56 wants to merge 2 commits into
Open
fix: improve icon-only button detection using tooltips and aria-labels#4977Oxygen56 wants to merge 2 commits into
Oxygen56 wants to merge 2 commits into
Conversation
Uses tooltip text and aria-label attributes to identify icon-only buttons that lack visible text content. Adds support for common tooltip data attributes (data-tooltip, data-original-title, data-bs-title, data-title) across detection, serialization, and history replay code paths. Fixes browser-use#4801 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Willow Lopez <100782273+Oxygen56@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enhances element detection for icon-only buttons using tooltip/aria-label text.
Problem
Icon-only buttons (e.g., a + button with only a tooltip saying "Create Test") were often missed by the browser agent. When a button lacks visible text content and has no aria-label, but has a tooltip (via HTML title attribute or JS tooltip library data attributes), the agent could not reliably identify or describe the element.
Changes
title,data-tooltip,data-original-title,data-bs-title,data-titleto icon detection attributes — small icon-sized elements with tooltip attributes are now detected as interactiveDEFAULT_INCLUDE_ATTRIBUTES(shown to LLM),STATIC_ATTRIBUTES(used for element hashing), andget_meaningful_text_for_llm()priority chainSAFE_ATTRIBUTESfor CSS selector generationtitleto history replay matching attributes and element error formattingtitleanddata-tooltipto click description key attributesFixes #4801
Summary by cubic
Detects icon-only buttons using tooltip and aria-label text so the browser agent stops missing them; supports common tooltip attributes across detection, serialization, and replay. Fixes #4801.
title,data-tooltip,data-original-title,data-bs-title,data-titleas interactive.titlein element matching and error messages.titleanddata-tooltipto improve action summaries.Written for commit b827366. Summary will update on new commits.