Skip to content

fix: improve icon-only button detection using tooltips and aria-labels#4977

Open
Oxygen56 wants to merge 2 commits into
browser-use:mainfrom
Oxygen56:fix/icon-button-tooltip-detection-4801
Open

fix: improve icon-only button detection using tooltips and aria-labels#4977
Oxygen56 wants to merge 2 commits into
browser-use:mainfrom
Oxygen56:fix/icon-button-tooltip-detection-4801

Conversation

@Oxygen56

@Oxygen56 Oxygen56 commented Jun 6, 2026

Copy link
Copy Markdown

Enhances element detection for icon-only buttons using tooltip/aria-label text.

Problem

Icon-only buttons (e.g., a + button with only a tooltip saying "Create Test") were often missed by the browser agent. When a button lacks visible text content and has no aria-label, but has a tooltip (via HTML title attribute or JS tooltip library data attributes), the agent could not reliably identify or describe the element.

Changes

  • clickable_elements.py: Added title, data-tooltip, data-original-title, data-bs-title, data-title to icon detection attributes — small icon-sized elements with tooltip attributes are now detected as interactive
  • views.py: Added tooltip data attributes to DEFAULT_INCLUDE_ATTRIBUTES (shown to LLM), STATIC_ATTRIBUTES (used for element hashing), and get_meaningful_text_for_llm() priority chain
  • service.py: Added tooltip data attributes to hidden element text fallback, pagination detection all_text, and pagination button text fallback
  • utils.py: Added tooltip data attributes to SAFE_ATTRIBUTES for CSS selector generation
  • agent/service.py: Added title to history replay matching attributes and element error formatting
  • tools/utils.py: Added title and data-tooltip to click description key attributes

Fixes #4801


Summary by cubic

Detects icon-only buttons using tooltip and aria-label text so the browser agent stops missing them; supports common tooltip attributes across detection, serialization, and replay. Fixes #4801.

  • Bug Fixes
    • Icon detection: treat small elements with title, data-tooltip, data-original-title, data-bs-title, data-title as interactive.
    • DOM serialization and selectors: include tooltip attrs in serialized attributes, element hashing, and safe attributes.
    • Text fallbacks and pagination: use tooltip attrs when deriving element text and when detecting pagination.
    • History replay and errors: include title in element matching and error messages.
    • Click descriptions: add title and data-tooltip to improve action summaries.

Written for commit b827366. Summary will update on new commits.

Review in cubic

Uses tooltip text and aria-label attributes to identify icon-only
buttons that lack visible text content. Adds support for common
tooltip data attributes (data-tooltip, data-original-title,
data-bs-title, data-title) across detection, serialization,
and history replay code paths.

Fixes browser-use#4801

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 6 files

Re-trigger cubic

Signed-off-by: Willow Lopez <100782273+Oxygen56@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

browser-use fails to identify/click icon-only "Create Test" button when tooltip is present but accessible label is missing

1 participant