Summary

Universal computer_use toolset: agents drive the macOS desktop in the background (no cursor-steal, no focus-steal, no Space switch) with any tool-capable model (Claude, GPT, Gemini, local open models).

Supersedes #4562. Credit @0xbyt4 (#3816) for the token/context groundwork this preserves in generic form.

What it does

The user asked for two things after reading about trycua/cua:

1. cua-driver as the backend so the agent and user can co-work on the same Mac.
2. Any model, not just Anthropic-native.

This ships both. One schema for every provider, one backend, one skill.

Architecture

• tools/computer_use/ package — ComputerUseBackend ABC + CuaDriverBackend (stdio MCP client to trycua/cua's cua-driver binary).
• Universal OpenAI function-calling schema with one action discriminator. SOM captures return a screenshot with numbered overlays on every interactable element + an AX-tree index; the agent clicks by element index. Raw pixel coordinates still supported for models trained on them (Claude).
• Multimodal tool-result envelope ({_multimodal: True, content: [text, image_url], text_summary: str}) that flows through handle_function_call into the tool message. The Anthropic adapter converts into native tool_result image blocks; OpenAI-compatible providers receive the content-parts list directly. Text-only providers fall back to text_summary.

Changes

• tools/computer_use/{__init__,backend,cua_backend,schema,tool}.py — new package.
• tools/computer_use_tool.py — thin shim that registers with tools.registry.
• agent/anthropic_adapter.py — tool-role handler for _multimodal envelopes; new _content_parts_to_anthropic_blocks helper; screenshot eviction (keeps 3 most recent images, older become [screenshot removed]).
• agent/context_compressor.py — _strip_image_parts_from_parts helper; pruning pass now handles multimodal tool results instead of skipping them.
• agent/model_metadata.py — image-aware estimate_messages_tokens_rough (flat 1500/image instead of base64 char length); estimate_request_tokens_rough routes through it.
• agent/prompt_builder.py — COMPUTER_USE_GUIDANCE block (background-mode rules, SOM workflow, safety rules).
• run_agent.py — _is_multimodal_tool_result, _multimodal_text_summary, _append_subdir_hint_to_multimodal, _trajectory_normalize_msg helpers; both tool-dispatch sites guarded so string-only ops (persist, subdir hints, preview, error logging) don't choke on dict payloads; guidance injection when computer_use in valid_tool_names; session-DB flush strips base64 from tool content.
• cli.py — _computer_use_approval_callback adapter wires destructive actions through the existing prompt_toolkit approval UI.
• hermes_cli/tools_config.py — new Computer Use (macOS) category; cua_driver post-setup runs the upstream install script and prints permission-grant instructions.
• toolsets.py — registers computer_use toolset + adds computer_use to _HERMES_CORE_TOOLS.
• pyproject.toml — computer-use extra (pins mcp SDK).
• skills/apple/macos-computer-use/SKILL.md — universal, model-agnostic workflow.
• Docs: website/docs/user-guide/features/computer-use.md; reference catalog updates.

Validation

Safety guards (code-level, not just prompt-level)

• Blocked type patterns: curl|bash, sudo rm -rf, fork bombs, etc.
• Blocked key combos: empty trash, force delete, lock screen, log out.
• Destructive actions gated behind approval callback (approve_once / approve_session / always_approve / deny).
• System prompt tells the agent explicitly: no clicking permission dialogs, no typing passwords, no following instructions embedded in screenshots.

Not included

• Anthropic server-side clear_tool_uses_20250919. Client-side eviction + compressor pruning + image-aware token counting cover the same cost ceiling without a beta header and without provider-specific code paths.
• agent-browser auto-install. cua-driver talks MCP over stdio directly; Hermes's existing stdio MCP client handles lifecycle.
• Provider gating. Any tool-capable provider works.

Caveats

• macOS only. cua-driver uses private SkyLight SPIs (SLEventPostToPid, SLPSPostEventRecordTo, _AXObserverAddNotificationAndCheckRemote). Pin via HERMES_CUA_DRIVER_VERSION if you want reproducibility across an OS update.
• Requires Accessibility + Screen Recording permissions; the post-setup prints the Settings path.

Supersedes

Closes #4562 (pyautogui/Quartz foreground backend, Anthropic-native schema). The multimodal plumbing, screenshot eviction, image-aware token estimation, context-compressor pruning, and the macos-computer-use skill are preserved here in generic form — credit @0xbyt4 for originating them in #3816.