Skip to content

fix: target explicit computer use app windows#25674

Open
helaskeutuja wants to merge 2 commits into
NousResearch:mainfrom
helaskeutuja:fix/computer-use-explicit-app-windows
Open

fix: target explicit computer use app windows#25674
helaskeutuja wants to merge 2 commits into
NousResearch:mainfrom
helaskeutuja:fix/computer-use-explicit-app-windows

Conversation

@helaskeutuja

Copy link
Copy Markdown

Summary

  • Fix explicit computer_use capture(app=...) targeting for macOS apps whose usable AX window is not returned by list_windows(on_screen_only=True)
  • Add normalized window selection and fallback to all windows for explicit app filters
  • Add a regression test covering the DaVinci Resolve-style off-screen main window case

Test Plan

  • scripts/run_tests.sh tests/tools/test_computer_use.py

Real behavior proof

  • Behavior or issue addressed: computer_use capture(app="DaVinci Resolve") could fall back to iTerm2 because DaVinci Resolve's main AX window appeared only when listing all windows, not on-screen-only windows.
  • Real environment tested: macOS M5 Max, DaVinci Resolve running locally, cua-driver 0.1.9, Accessibility and Screen Recording granted.
  • Exact steps or command run after this patch: CUA_DRIVER_MCP_NO_RELAUNCH=1 hermes chat -q 'Käytä vain computer_use-työkalua. Tee capture mode=ax app="DaVinci Resolve"...' --toolsets computer_use -Q
  • Evidence after fix: Tool result reported App/window: DaVinci Resolve, “Untitled Project” and Elementtimäärä: 104.
  • Observed result after fix: Explicit app targeting selected DaVinci Resolve instead of iTerm2. A mode=som read-only capture also selected DaVinci Resolve and returned 104 interactable elements, but no screenshot dimensions.
  • What was not tested: No clicks, typing, focus changes, or Resolve UI mutations were performed.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/tools Tool registry, model_tools, toolsets labels May 14, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related to #24919 which addresses the same CUA window targeting issue (both fix fallback to wrong frontmost window). Part of the #24170 bug cluster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants