fix(computer-use): surface app=… filter no-match instead of silently using frontmost (#24170 bug 1)#24324
Closed
briandevans wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a correctness issue in the macOS cua-driver backend where capture(app=...) and focus_app(app=...) would silently fall back to the frontmost window when the requested app filter matched no running app (commonly due to macOS returning localized app names from list_windows). The change ensures the no-match case is explicitly surfaced so the agent doesn’t interact with the wrong application.
Changes:
- Update
CuaDriverBackend.capture()to return an explicit emptyCaptureResultwith a diagnosticwindow_titlewhenapp=matches no windows (instead of capturing the frontmost window). - Update
CuaDriverBackend.focus_app()to fail (ok=False) whenapp=matches no windows (instead of selecting the frontmost window). - Add regression tests covering both behaviors for match and no-match cases.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tools/computer_use/cua_backend.py | Prevent silent fallback to the frontmost window when an app= filter matches nothing in capture() and focus_app(). |
| tests/tools/test_computer_use.py | Add regression tests ensuring app= no-match is surfaced (capture returns empty + diagnostic; focus_app returns not-ok). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
19 tasks
62ef46e to
88017a0
Compare
…using frontmost (NousResearch#24170 bug 1) `CuaDriverBackend.capture(app=X)` and `focus_app(app=X)` silently fell back to the frontmost on-screen window when X matched no app — typically a menu-bar utility (e.g. "Fuwari" in the bug reporter's case) rather than the requested app. The agent then received UI elements for the wrong app and clicked / typed into it. The root cause is a localized macOS app name mismatch: `list_windows` returns the localized `app_name` (e.g. "計算機" on a Japanese/Chinese system) but callers naturally pass the English name ("Calculator"). The substring filter doesn't match, and the code falls through to picking the frontmost window with no signal that the filter was effectively dropped. Fix: - `capture(app=…)`: when the filter matches nothing, return a `CaptureResult` with empty `app`/`elements` and a diagnostic `window_title` pointing the caller at `list_apps` and noting the localized-name convention. `_active_pid` / `_active_window_id` are left untouched so a subsequent action doesn't inadvertently hit the wrong process. - `focus_app(app=…)`: when the filter matches nothing, set `target = None` and let the existing `return ActionResult(ok=False, …, "No on-screen window found for app …")` path fire instead of falsely reporting success on the frontmost window. This addresses bug 1 only from NousResearch#24170. Bugs 2 & 5 are addressed in NousResearch#24242, bugs 3 & 4 in NousResearch#24181. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
88017a0 to
9be35a4
Compare
This was referenced May 21, 2026
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
CuaDriverBackend.capture(app=X)andfocus_app(app=X)silently fell back to the frontmost on-screen window when X matched no app — typically a menu-bar utility (e.g. "Fuwari" in the bug reporter's case) rather than the requested app. The agent then received UI elements for the wrong app and clicked/typed into it.list_windowsreturns the macOS-localizedapp_name(e.g."計算機"on a Chinese/Japanese system) but callers naturally pass the English name ("Calculator"). The substring filter doesn't match, and the code falls through —capture()silently keeps all windows;focus_app()silently uses the frontmost. The agent receives UI elements for the wrong window with no signal that theapp=filter was effectively dropped.The fix:
capture(app=…)returns aCaptureResultwith emptyapp/elementsand a diagnosticwindow_titlepointing the caller atlist_appsand noting the localized-name convention._active_pid/_active_window_idare left untouched.focus_app(app=…)setstarget = Noneso the existingActionResult(ok=False, "No on-screen window found for app …")path fires instead of falsely reporting success on the frontmost window. 26 prod lines (mostly comments), 134 test lines, 2 files. No cross-locale name resolution attempted — that would be a larger change;list_appsalready exposes the localized names the caller needs.Related Issue
Fixes bug 1 from #24170
Type of Change
Changes Made
tools/computer_use/cua_backend.py—capture(app=…)returns emptyCaptureResultwith diagnosticwindow_titlewhen filter matches nothing;focus_app(app=…)setstarget = Noneto surface the existingok=Falsepath.tests/tools/test_computer_use.py—TestCaptureAppFilterNoMatch(3 cases) +TestFocusAppFilterNoMatch(2 cases).How to Test
uv run --with pytest --with pytest-xdist --with pytest-asyncio python3 -m pytest tests/tools/test_computer_use.py -vAssertionError: 'Fuwari' == ''). Restore — green.Checklist
Code
fix(scope):,feat(scope):, etc.)Documentation & Housekeeping
docs/, docstrings) — N/Acli-config.yaml.exampleif I added/changed config keys — N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — N/ARelated / Positioning
Addresses bug 1 from #24170. Bugs 2 & 5 are addressed in #24242 (Bartok9); bugs 3 & 4 in #24181 (liuhao1024).
type_text_chars→type_text; implementdragtype_text_charsrename_last_appto preserve app for capture_after;_ELEMENT_LINE_REfor v0.1.6id=Labelformatapp=filter — orthogonal to Bartok9's_last_appchange, no merge conflict expectedBartok9's PR body explicitly notes that bug 1 is tracked separately.