feat(tui): subagent spawn observability overlay by OutThisLife · Pull Request #14045 · NousResearch/hermes-agent

OutThisLife · 2026-04-22T15:39:02Z

Summary

A /agents overlay that turns recursive delegate_task fan-out into a first-class audit surface — live monitoring, per-branch cost/token/file rollups, kill + pause controls, and post-hoc replay + diff. Solves the "uncapped depth/width is hard to audit" problem.

Reference implementations (cc swarm, opencode, openclaw) were surveyed — none ship nested side-by-side scrollable panes inside an Ink overlay, so this uses view switching (list → detail) which is the only shape Ink's Yoga flex handles cleanly at depth.

What's in the box

Python (delegate_tool + tui_gateway)

Every subagent.* event now carries subagent_id, parent_id, depth, model, tool_count
subagent.complete additionally ships tokens (in/out/reasoning), cost_usd, api_calls, files_read/written, and a tool-call output tail
New subagent.spawn_requested event fires pre-run so queued children show up instantly
_active_subagents registry + pause flag expose per-branch kill + global pause
New RPCs: delegation.status, delegation.pause, subagent.interrupt, spawn_tree.save/list/load (disk persistence under $HERMES_HOME/spawn-trees)

TUI (/agents overlay)

List mode — full-width picker with Gantt strip above; wall-clock tick ruler + per-bar duration labels; heat-coloured marker per hot branch; trailing slot shows most-recent tool label for running rows (timeline carries the live clock — no duplicated timers).
Detail mode — full-width scrollable pane with accordion sections (Budget / Files / Tool calls / Output / Progress / Summary). Accordion open-state lives on a shared atom so it survives cursor + mode changes.
Mouse — wheel scrolls detail pane / moves list cursor; OverlayScrollbar supports click-to-jump + click-drag on the thumb (parity with TranscriptScrollbar).
Keyboard: j/k navigate · Enter/→ open detail · Esc/← back · [/] history nav · x kill · X kill subtree · p pause · s sort · f filter · q close.
Auto-follow: when a live turn ends, the overlay slides onto the just-archived snapshot and relabels the header (Last turn · finished HH:MM:SS). Late subagent.complete/tool/progress events are update-only so they can't resurrect a finished subagent into turn.subagents and block the auto-follow.
/replay [N|last|list|load <path>] — walks in-memory + disk history.
/replay-diff <a> <b> — side-by-side tree compare with Δ rollups.

Status bar HUD

SpawnHud renders depth/concurrency vs caps with warn/error escalation as the tree approaches limits.

Theme

DARK dim bumped #B8860B → #CC9B1F for readable secondary text (LIGHT unchanged — it was already high-contrast).
Overlay uses the full theme role hierarchy: cornsilk body / label field names / dim bullets / amber accents — zero hardcoded hex.

Shared helpers (ui-tui/src/lib/subagentTree)

fmtDuration(seconds) — now exported; single source for the Ns / Nm Ss labels used by the Gantt timeline, list rows, and formatSummary. Separate from domain/messages.fmtDuration(ms) which keeps the ms contract.
topLevelSubagents(items) — same orphan-promoting rule as buildSubagentTree; reused by the disk snapshot label, diff pane, and prior hand-rolled filters.

Test plan

Notes

Kill is cooperative (thread interrupt at iteration boundary), same mechanism as turn-level Ctrl+C. Branches take up to a few seconds to transition to interrupted state after x.
Event schema additions are backward-compatible — older gateway versions that omit depth / parent_id still render flat in the overlay.
Snapshot disk layout is $HERMES_HOME/spawn-trees/<session_id>/<timestamp>.json, plus a _index.jsonl sidecar for cheap listing.

Adds a live + post-hoc audit surface for recursive delegate_task fan-out. None of cc/oc/oclaw tackle nested subagent trees inside an Ink overlay; this ships a view-switched dashboard that handles arbitrary depth + width. Python - delegate_tool: every subagent event now carries subagent_id, parent_id, depth, model, tool_count; subagent.complete also ships input/output/ reasoning tokens, cost, api_calls, files_read/files_written, and a tail of tool-call outputs - delegate_tool: new subagent.spawn_requested event + _active_subagents registry so the overlay can kill a branch by id and pause new spawns - tui_gateway: new RPCs delegation.status, delegation.pause, subagent.interrupt, spawn_tree.save/list/load (disk under \$HERMES_HOME/spawn-trees/<session>/<ts>.json) TUI - /agents overlay: full-width list mode (gantt strip + row picker) and Enter-to-drill full-width scrollable detail mode; inverse+amber selection, heat-coloured branch markers, wall-clock gantt with tick ruler, per-branch rollups - Detail pane: collapsible accordions (Budget, Files, Tool calls, Output, Progress, Summary); open-state persists across agents + mode switches via a shared atom - /replay [N|last|list|load <path>] for in-memory + disk history; /replay-diff <a> <b> for side-by-side tree comparison - Status-bar SpawnHud warns as depth/concurrency approaches caps; overlay auto-follows the just-finished turn onto history[1] - Theme: bump DARK dim #B8860B → #CC9B1F for readable secondary text globally; keep LIGHT untouched Tests: +29 new subagentTree unit tests; 215/215 passing.

The Write tool that wrote the cleaned overlay split the `if` keyword across two lines in 9 places (` i\nf (cond) {`), which silently passed one typecheck run but actually left the handler as broken JS — every keystroke threw. Input froze in the /agents overlay (j/k/arrows/q/etc. all no-ops) while the 500ms now-tick kept rendering, so the UI looked "frozen but the timeline moves". Reflows the handler as-intended with no behaviour change.

- createGatewayEventHandler: remove dead `return` after a block that always returns (tool.complete case). The inner block exits via both branches so the outer statement was never reachable. Was pre-existing on main; fixed here because it was the only thing blocking `npm run fix` on this branch. - agentsOverlay + ops: prettier reformatting. `npm run fix` / `npm run type-check` / `npm test` all clean.

Copilot

Pull request overview

Adds a new spawn-tree observability surface for recursive delegate_task fan-out, including live monitoring and controls in the TUI (/agents), plus gateway/Python plumbing for richer subagent event metadata and persistence of completed spawn trees for replay/diff.

Changes:

Introduces spawn-tree reconstruction + aggregation utilities and test coverage (subagentTree.ts + subagentTree.test.ts).
Adds a full-screen /agents overlay (list/detail modes, timeline, kill/pause, history/replay/diff) and a status-bar HUD.
Extends gateway/Python event/RPC schemas for subagent identity, rollups, and disk-backed snapshot persistence (spawn_tree.*, delegation.*, subagent.interrupt).

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
ui-tui/src/types.ts	Extends TUI domain types for subagent identity/rollups and delegation status.
ui-tui/src/theme.ts	Adjusts dark-theme `dim` (and matching session label/border) for readability.
ui-tui/src/lib/subagentTree.ts	New pure utilities to build a spawn tree, aggregate metrics, and format summaries/sparklines.
ui-tui/src/gatewayTypes.ts	Extends subagent event payload typing and adds delegation/snapshot RPC response types.
ui-tui/src/components/thinking.tsx	Renders spawn-tree rollups inline and adds hotness-based stem coloring in subagent accordions.
ui-tui/src/components/appLayout.tsx	Switches main pane to the Agents overlay when enabled; hides prompt/composer while overlay is open.
ui-tui/src/components/appChrome.tsx	Adds a `SpawnHud` in the status bar showing depth/concurrency vs caps and paused state.
ui-tui/src/components/agentsOverlay.tsx	New `/agents` full-screen overlay: list/detail, gantt strip, kill/pause, history nav, replay diff.
ui-tui/src/app/useInputHandlers.ts	Ensures Ctrl+C closes the agents overlay (like other overlays).
ui-tui/src/app/turnController.ts	Adds in-memory snapshot push on turn completion and supports fire-and-forget disk persistence callback.
ui-tui/src/app/spawnHistoryStore.ts	New nanostore for spawn snapshot history + diff pairing; includes disk-load normalization.
ui-tui/src/app/slash/commands/ops.ts	Adds `/agents`, `/replay`, `/replay-diff` slash commands integrating overlay + disk RPCs.
ui-tui/src/app/overlayStore.ts	Adds overlay state for agents + “flow-only” overlay reset to avoid auto-closing `/agents` on turn end.
ui-tui/src/app/interfaces.ts	Extends OverlayState interface with agents overlay fields.
ui-tui/src/app/delegationStore.ts	New nanostore for delegation caps/paused and shared overlay accordion open-state.
ui-tui/src/app/createGatewayEventHandler.ts	Wires new `subagent.spawn_requested` event and injects spawn-tree disk persistence via RPC.
ui-tui/src/tests/subagentTree.test.ts	Comprehensive unit tests for tree building/aggregation/formatting/hotness.
tui_gateway/server.py	Adds delegation control + snapshot RPC methods and relays new subagent identity/rollup fields.
tools/delegate_tool.py	Adds pause flag + active-subagent registry + emits richer subagent lifecycle/rollup events.

Comments suppressed due to low confidence (1)

tools/delegate_tool.py:1166

delegate_task emits subagent.complete with status="timeout" / "error", but the TUI type + renderers only recognize completed|failed|interrupted|queued|running. If a timeout occurs, the client can end up with an unknown status and crash on lookups like STATUS_GLYPH/STATUS_RANK. Prefer mapping these terminal states onto the existing schema (e.g., treat timeout/error as failed and add a separate exit_reason field), or update the UI types + glyph/rank tables to handle the new statuses consistently.

                    child_progress_cb(
                        "subagent.complete",
                        preview=(
                            f"Timed out after {duration}s"
                            if is_timeout
                            else str(_timeout_exc)
                        ),
                        status="timeout" if is_timeout else "error",
                        duration_seconds=duration,
                        summary="",
                    )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

alt-glitch · 2026-04-22T15:55:35Z

Related to #5513 (subagent control panel), #5625 (workload status bar), and #6594 (TUI dashboards for delegation). This is the most comprehensive implementation attempt.

Four real issues Copilot flagged: 1. delegate_tool: `_build_child_agent` never passed `toolsets` to the progress callback, so the event payload's `toolsets` field (wired through every layer) was always empty and the overlay's toolsets row never populated. Thread `child_toolsets` through. 2. event handler: the race-protection on subagent.spawn_requested / subagent.start only preserved `completed`, so a late-arriving queued event could clobber `failed` / `interrupted` too. Preserve any terminal status (`completed | failed | interrupted`). 3. SpawnHud: comment claimed concurrency was approximated by "widest level in the tree" but code used `totals.activeCount` (total across all parents). `max_concurrent_children` is a per-parent cap, so activeCount over-warns for multi-orchestrator runs. Switch to `max(widthByDepth(tree))`; the label now reads `⚡W/cap+extra` where W is the widest level (drives the ratio) and `+extra` is the rest. 4. spawn_tree.list: comment said "peek header without parsing full list" but the code json.loads()'d every snapshot. Adds a per-session `_index.jsonl` sidecar written on save; list() reads only the index (with a full-scan fallback for pre-index sessions). O(1) per snapshot now vs O(file-size).

alt-glitch · 2026-04-22T15:56:39Z

Related to #5513, #5625, #6594.

- List rows: pad the status dot with space before (heat-marker gap or matching 2-space filler) and after (3 spaces to goal) so `●` / `○` / `✓` / `■` / `✗` don't read glued to the heat bar or the goal text. - Gantt rows: bump id→bar separator from 1 to 2 spaces; widen the id gutter from 4 to 5 cols and re-align the ruler lead to match.

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- delegate_task: use shared tool_error() for the paused-spawn early return so the error envelope matches the rest of the tool. - Disk snapshot label: treat orphaned nodes (parentId missing from the snapshot) as top-level, matching buildSubagentTree / summarizeLabel.

…rs via lib Pull duplicated rules into ui-tui/src/lib/subagentTree so the live overlay, disk snapshot label, and diff pane all speak one dialect: - export fmtDuration(seconds) — was a private helper in subagentTree; agentsOverlay's local secLabel/fmtDur/fmtElapsedLabel now wrap the same core (with UI-only empty-string policy). - export topLevelSubagents(items) — matches buildSubagentTree's orphan semantics (no parent OR parent not in snapshot). Replaces three hand- rolled copies across createGatewayEventHandler (disk label), agentsOverlay DiffPane, and prior inline filters. Also collapse agentsOverlay boilerplate: - replace IIFE title + inner `delta` helper with straight expressions; - introduce module-level diffMetricLine for replay-diff rows; - tighten OverlayScrollbar (single thumbColor expression, vBar/thumbBody). Adds unit coverage for the new exports (fmtDuration + topLevelSubagents). No behaviour change; 221 tests pass.

…ers (#17727) Covers ~60 merged PRs from Apr 15–29 that shipped user-visible behavior without docs coverage. No functional code changes; docs + static manifest regeneration only. Highlights: Stale / incorrect: - configuration.md: auxiliary auto-routing line was wrong since #11900; now correctly states auto routes to the main model, with a note on the cost trade-off and per-task override pattern. - integrations/providers.md + configuration.md compression intro: removed stale 'Gemini Flash via OpenRouter' claim. - website/static/api/model-catalog.json: rebuilt from hermes_cli/models.py so the live manifest picks up tencent/hy3-preview (and remains in sync for future model-catalog PRs). Platform messaging (#17417 #16997 #16193 #14315 #13151 #11794 #10610 #10283 #10246 #11564 #13178): - Signal: native formatting (bodyRanges), reply quotes, reactions. - Telegram: table rendering (bullets + code-block fallback), disable_link_previews, group_allowed_chats. - Slack: strict_mention config. - Discord: slash_commands disable, send_animation GIF, send_message native media attachments. - DingTalk: require_mention + allowed_users. CLI (#16052 #16539 #16566 #15841 #14798 #10043): - New 'hermes fallback' interactive manager. - New 'hermes update --check', '--backup' flag, and pre-update pairing snapshot behavior. - 'hermes gateway start/restart --all' multi-profile flag. - cron.md: 'hermes tools' as a platform, per-job enabled_toolsets, wakeAgent gate, context_from chaining. Config keys / env vars (#17305 #17026 #17000 #15077 #14557 #14227 #14166 #14730 #17008): - terminal.docker_run_as_host_user, display.runtime_metadata_footer, compression.hygiene_hard_message_limit, HINDSIGHT_TIMEOUT, skills.guard_agent_created, TAVILY_BASE_URL, security.allow_private_urls, agent.api_max_retries, gateway hot-reload of compression/context_length config edits. TUI / CLI UX (#17130 #17113 #17175 #17150 #16707 #12312 #12305 #12934 #14810 #14045 #17286 #17126): - HERMES_TUI_RESUME, HERMES_TUI_THEME, LaTeX rendering, busy-indicator styles, ctrl-x queued-message delete, git branch in status bar, per- prompt elapsed stopwatch, external-editor keybind, markdown stripping, TUI voice-mode parity, /agents overlay, /reload + /mouse. Gateway features (#16506 #15027 #13428 #12116): - Native multimodal image routing based on vision capability. - /usage account-limits section. - /steer slash command (added to reference + explanation in CLI). Plugins / hooks (#12929 #12972 #10763 #16364): - transform_tool_result, transform_terminal_output plugin hooks. - PluginContext.dispatch_tool() documented with slash-command example. - google_meet bundled plugin entry under built-in-plugins.md. Other (#16576 #16572 #16383 #15878 #15608 #15606 #14809 #14767 #14231 #14232 #14307 #13683 #12373 #11891 #11291 #10066): - hermes backup exclusions (WAL/SHM/journal + checkpoints/). - security.md hardline blocklist (floor below --yolo). - FHS install layout for root installs. - openssh-client + docker-cli baked into the Docker image. - MEDIA: tag supported extensions table (docs/office/archives/pdf). - Remote-to-host file sync on SSH/Modal/Daytona teardown. - 'hermes model' -> Configure Auxiliary Models interactive picker. - Podman support via HERMES_DOCKER_BINARY. Providers / STT / one-shot (#15045 #14473 #15704): - alibaba-coding-plan first-class provider entry. - xAI Grok STT as a 6th transcription option. - 'hermes -z' scripted one-shot mode + HERMES_INFERENCE_MODEL. Build: 'docusaurus build' succeeds. No new broken links/anchors; pre-existing warnings unchanged.

Four real issues Copilot flagged: 1. delegate_tool: `_build_child_agent` never passed `toolsets` to the progress callback, so the event payload's `toolsets` field (wired through every layer) was always empty and the overlay's toolsets row never populated. Thread `child_toolsets` through. 2. event handler: the race-protection on subagent.spawn_requested / subagent.start only preserved `completed`, so a late-arriving queued event could clobber `failed` / `interrupted` too. Preserve any terminal status (`completed | failed | interrupted`). 3. SpawnHud: comment claimed concurrency was approximated by "widest level in the tree" but code used `totals.activeCount` (total across all parents). `max_concurrent_children` is a per-parent cap, so activeCount over-warns for multi-orchestrator runs. Switch to `max(widthByDepth(tree))`; the label now reads `⚡W/cap+extra` where W is the widest level (drives the ratio) and `+extra` is the rest. 4. spawn_tree.list: comment said "peek header without parsing full list" but the code json.loads()'d every snapshot. Adds a per-session `_index.jsonl` sidecar written on save; list() reads only the index (with a full-scan fallback for pre-index sessions). O(1) per snapshot now vs O(file-size).

- delegate_task: use shared tool_error() for the paused-spawn early return so the error envelope matches the rest of the tool. - Disk snapshot label: treat orphaned nodes (parentId missing from the snapshot) as top-level, matching buildSubagentTree / summarizeLabel.

…bservability feat(tui): subagent spawn observability overlay

Four real issues Copilot flagged: 1. delegate_tool: `_build_child_agent` never passed `toolsets` to the progress callback, so the event payload's `toolsets` field (wired through every layer) was always empty and the overlay's toolsets row never populated. Thread `child_toolsets` through. 2. event handler: the race-protection on subagent.spawn_requested / subagent.start only preserved `completed`, so a late-arriving queued event could clobber `failed` / `interrupted` too. Preserve any terminal status (`completed | failed | interrupted`). 3. SpawnHud: comment claimed concurrency was approximated by "widest level in the tree" but code used `totals.activeCount` (total across all parents). `max_concurrent_children` is a per-parent cap, so activeCount over-warns for multi-orchestrator runs. Switch to `max(widthByDepth(tree))`; the label now reads `⚡W/cap+extra` where W is the widest level (drives the ratio) and `+extra` is the rest. 4. spawn_tree.list: comment said "peek header without parsing full list" but the code json.loads()'d every snapshot. Adds a per-session `_index.jsonl` sidecar written on save; list() reads only the index (with a full-scan fallback for pre-index sessions). O(1) per snapshot now vs O(file-size).

- delegate_task: use shared tool_error() for the paused-spawn early return so the error envelope matches the rest of the tool. - Disk snapshot label: treat orphaned nodes (parentId missing from the snapshot) as top-level, matching buildSubagentTree / summarizeLabel.

…bservability feat(tui): subagent spawn observability overlay

…ers (NousResearch#17727) Covers ~60 merged PRs from Apr 15–29 that shipped user-visible behavior without docs coverage. No functional code changes; docs + static manifest regeneration only. Highlights: Stale / incorrect: - configuration.md: auxiliary auto-routing line was wrong since NousResearch#11900; now correctly states auto routes to the main model, with a note on the cost trade-off and per-task override pattern. - integrations/providers.md + configuration.md compression intro: removed stale 'Gemini Flash via OpenRouter' claim. - website/static/api/model-catalog.json: rebuilt from hermes_cli/models.py so the live manifest picks up tencent/hy3-preview (and remains in sync for future model-catalog PRs). Platform messaging (NousResearch#17417 NousResearch#16997 NousResearch#16193 NousResearch#14315 NousResearch#13151 NousResearch#11794 NousResearch#10610 NousResearch#10283 NousResearch#10246 NousResearch#11564 NousResearch#13178): - Signal: native formatting (bodyRanges), reply quotes, reactions. - Telegram: table rendering (bullets + code-block fallback), disable_link_previews, group_allowed_chats. - Slack: strict_mention config. - Discord: slash_commands disable, send_animation GIF, send_message native media attachments. - DingTalk: require_mention + allowed_users. CLI (NousResearch#16052 NousResearch#16539 NousResearch#16566 NousResearch#15841 NousResearch#14798 NousResearch#10043): - New 'hermes fallback' interactive manager. - New 'hermes update --check', '--backup' flag, and pre-update pairing snapshot behavior. - 'hermes gateway start/restart --all' multi-profile flag. - cron.md: 'hermes tools' as a platform, per-job enabled_toolsets, wakeAgent gate, context_from chaining. Config keys / env vars (NousResearch#17305 NousResearch#17026 NousResearch#17000 NousResearch#15077 NousResearch#14557 NousResearch#14227 NousResearch#14166 NousResearch#14730 NousResearch#17008): - terminal.docker_run_as_host_user, display.runtime_metadata_footer, compression.hygiene_hard_message_limit, HINDSIGHT_TIMEOUT, skills.guard_agent_created, TAVILY_BASE_URL, security.allow_private_urls, agent.api_max_retries, gateway hot-reload of compression/context_length config edits. TUI / CLI UX (NousResearch#17130 NousResearch#17113 NousResearch#17175 NousResearch#17150 NousResearch#16707 NousResearch#12312 NousResearch#12305 NousResearch#12934 NousResearch#14810 NousResearch#14045 NousResearch#17286 NousResearch#17126): - HERMES_TUI_RESUME, HERMES_TUI_THEME, LaTeX rendering, busy-indicator styles, ctrl-x queued-message delete, git branch in status bar, per- prompt elapsed stopwatch, external-editor keybind, markdown stripping, TUI voice-mode parity, /agents overlay, /reload + /mouse. Gateway features (NousResearch#16506 NousResearch#15027 NousResearch#13428 NousResearch#12116): - Native multimodal image routing based on vision capability. - /usage account-limits section. - /steer slash command (added to reference + explanation in CLI). Plugins / hooks (NousResearch#12929 NousResearch#12972 NousResearch#10763 NousResearch#16364): - transform_tool_result, transform_terminal_output plugin hooks. - PluginContext.dispatch_tool() documented with slash-command example. - google_meet bundled plugin entry under built-in-plugins.md. Other (NousResearch#16576 NousResearch#16572 NousResearch#16383 NousResearch#15878 NousResearch#15608 NousResearch#15606 NousResearch#14809 NousResearch#14767 NousResearch#14231 NousResearch#14232 NousResearch#14307 NousResearch#13683 NousResearch#12373 NousResearch#11891 NousResearch#11291 NousResearch#10066): - hermes backup exclusions (WAL/SHM/journal + checkpoints/). - security.md hardline blocklist (floor below --yolo). - FHS install layout for root installs. - openssh-client + docker-cli baked into the Docker image. - MEDIA: tag supported extensions table (docs/office/archives/pdf). - Remote-to-host file sync on SSH/Modal/Daytona teardown. - 'hermes model' -> Configure Auxiliary Models interactive picker. - Podman support via HERMES_DOCKER_BINARY. Providers / STT / one-shot (NousResearch#15045 NousResearch#14473 NousResearch#15704): - alibaba-coding-plan first-class provider entry. - xAI Grok STT as a 6th transcription option. - 'hermes -z' scripted one-shot mode + HERMES_INFERENCE_MODEL. Build: 'docusaurus build' succeeds. No new broken links/anchors; pre-existing warnings unchanged.