feat: eager inbox delivery for providers that buffer input during processing#251
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #251 +/- ##
=======================================
Coverage ? 92.29%
=======================================
Files ? 69
Lines ? 6540
Branches ? 0
=======================================
Hits ? 6036
Misses ? 504
Partials ? 0
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
dea1af6 to
0777ac9
Compare
There was a problem hiding this comment.
Pull request overview
Adds an opt-in “eager inbox delivery” mode to reduce inter-turn latency for providers that can buffer input while processing (notably Claude Code), plus several Claude Code–specific robustness updates.
Changes:
- Introduces
CAO_EAGER_INBOX_DELIVERYgating and provider capability signaling (accepts_input_while_processing) to allow inbox delivery duringPROCESSING/WAITING_USER_ANSWERfor eligible providers. - Updates inbox watchdog logic to optionally skip log “idle pattern” checks for eager-capable providers.
- Enhances Claude Code provider behavior (agent profile fallback routing, env-var preservation, response extraction prompt anchoring) and adds/updates unit tests around the new inbox behavior and profile fallback.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/cli_agent_orchestrator/services/inbox_service.py |
Implements eager eligibility logic in check_and_send_pending_messages() and adjusts watchdog path to optionally bypass idle-pattern checks. |
src/cli_agent_orchestrator/providers/claude_code.py |
Adds eager capability via _initialized, adds agent routing behavior, adjusts env unsetting, and tightens response extraction stop conditions. |
src/cli_agent_orchestrator/providers/base.py |
Adds accepts_input_while_processing capability flag to the base provider interface (default False). |
src/cli_agent_orchestrator/constants.py |
Adds EAGER_INBOX_DELIVERY constant derived from CAO_EAGER_INBOX_DELIVERY. |
test/services/test_inbox_service.py |
Adds unit tests covering eager-delivery state/capability combinations and watchdog behavior. |
test/providers/test_claude_code_unit.py |
Updates initialization test to validate missing-profile fallback to --agent <name>. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Route based on profile state | ||
| native = getattr(profile, "native_agent", None) if profile else None | ||
| if profile is not None and isinstance(native, str) and native: | ||
| # Thin wrapper: CAO profile maps to a native Claude Code agent. | ||
| # Let Claude Code handle all config (MCP servers, hooks, tools, model). | ||
| # CAO_TERMINAL_ID propagates via tmux pane env inheritance. | ||
| command_parts.extend(["--agent", native]) |
| # Extract everything after the last ⏺ until: | ||
| # 1. A start-of-line idle prompt (❯ or >) — the definitive boundary | ||
| # 2. A completion stat line ("✻ Sautéed for 14s") — trims the stat | ||
| # Using start-of-line anchor avoids false stops on ">" inside | ||
| # response content (Java generics, git diffs, HTML tags, etc.). | ||
| remaining_text = script_output[start_pos:] | ||
|
|
||
| # Split by lines and extract response | ||
| lines = remaining_text.split("\n") | ||
| response_lines = [] | ||
|
|
||
| for line in lines: | ||
| # Stop at next > prompt or separator line | ||
| if re.match(r">\s", line) or "────────" in line: | ||
| clean_line = re.sub(ANSI_CODE_PATTERN, "", line).strip() | ||
| if self._SOL_IDLE_RE.match(line): | ||
| break | ||
| if "────────" in line: | ||
| break |
| @patch("cli_agent_orchestrator.services.inbox_service.terminal_service") | ||
| @patch("cli_agent_orchestrator.services.inbox_service.provider_manager") | ||
| @patch("cli_agent_orchestrator.services.inbox_service.get_pending_messages") | ||
| def test_delivery_waiting_user_answer_with_eager_enabled_and_capable_provider( |
There was a problem hiding this comment.
I haven't been using claude for the past month so they may have changed this, but want to double check that this is actually the intended case.
Last time i was testing similar eager inbox delivery mechanism on WAITING_USER_ANSWER case, it would not work
There was a problem hiding this comment.
yeah, that is why I tested it and it being working fine with me.
| # Start-of-line idle prompt for extraction: ❯ or > at the beginning of a line | ||
| # (after optional ANSI codes). Mid-line ">" in Java generics, git diffs, HTML | ||
| # etc. must NOT trigger the stop condition. | ||
| _SOL_IDLE_RE = re.compile( |
There was a problem hiding this comment.
i would challenge why introducing new regex is necessary, since the motive here is to reduce reliance on regexp for IDLE status matching in the first place.
There was a problem hiding this comment.
What do you recommend instead ?
Current detecting I have noticed even fail with generics with java so I had to fix it to make sure it consistently works in all the cases.
| # PROCESSING or WAITING_USER_ANSWER state for providers that declare | ||
| # accepts_input_while_processing=True. Eliminates latency between agent turns | ||
| # for capable providers (e.g., Claude Code). | ||
| EAGER_INBOX_DELIVERY = os.environ.get("CAO_EAGER_INBOX_DELIVERY", "false").lower() == "true" |
There was a problem hiding this comment.
i'm an advocate for having this be default behavior instead of opt-in env-var. I guess the question is do we see any breaking change for existing workflows if this is enabled silently
There was a problem hiding this comment.
I was thinking about this. But since the prompt would land while processing it may not work for some prompt where you immediately asking it to summarize but not the end. So, made it optional. Later changes, we could remove it.
|
Overall I think this is a great change and I am doing something similar in my own cao build. However I would challenge the scoping of this PR, specifically for introduction of the native agent frontmatter field and |
|
should include documentation regarding the |
I just added as a part claude code support improvements. Agent was optional field if anyone is using it. So, make it like a single PR. |
Sure, I will add some documentations. |
0777ac9 to
14734c7
Compare
| @property | ||
| def accepts_input_while_processing(self) -> bool: | ||
| """Claude Code's Ink TUI buffers pasted input during processing. | ||
|
|
||
| Only true after initialization completes — during startup the REPL | ||
| isn't ready to accept input even though get_status() sees PROCESSING. | ||
| """ | ||
| return self._initialized |
|
@anilkmr-a2z looks good to me, can you please help to resolve conflicts. |
haofeif
left a comment
There was a problem hiding this comment.
Feel free to merge after you resolve conflicts.
…cessing Add opt-in eager delivery mode (CAO_EAGER_INBOX_DELIVERY env var) that allows inbox messages to be delivered to terminals in PROCESSING or WAITING_USER_ANSWER states, eliminating the latency gap between agent turns for capable providers. Changes: - Add EAGER_INBOX_DELIVERY constant gated behind CAO_EAGER_INBOX_DELIVERY - Add accepts_input_while_processing property to BaseProvider (default False) - Override in ClaudeCodeProvider (returns True only after initialization) - Relax status gate in check_and_send_pending_messages() with two-flag check (env var + provider capability) - Skip idle-pattern pre-check in watchdog for eager-capable providers - Add native_agent thin-wrapper routing: missing CAO profiles fall back to --agent <name> for Claude Code's native agent store - Add 9 unit tests for eager delivery + native agent fallback test Misc Claude Code provider fixes: - Preserve CLAUDE_CODE_EFFORT_LEVEL through CAO's env-unset command - Fix response extraction false stops on ">" in content (Java generics, git diffs, HTML) by using start-of-line anchor (_SOL_IDLE_RE)
14734c7 to
65800b1
Compare
Bring the event-driven architecture branch up to date with main (98 commits) and reconcile the rewrite with features that landed after it forked: eager inbox delivery (awslabs#251), the OpenCode poller, env-var forwarding (awslabs#259), memory curation (awslabs#254/awslabs#262), CORS auto-derive (awslabs#261), DNS host validation (awslabs#124), and the self-send guard (awslabs#24). Highlights: - Providers adopt the async initialize() + get_status(buffer) contract; copilot_cli/opencode_cli converted; kiro keeps colour-only ANSI stripping so carriage-return-redraw permission prompts aren't misread as idle. - Event-driven InboxService.deliver_pending with the awslabs#251 eager gate and message-sender attribution; OpenCode poller retained as a status-driven method; the watchdog (PollingObserver/LogFileHandler) is removed. - terminal_service.create_terminal is async (FIFO + StatusMonitor wiring); session_service.create_session, flow_service.execute_flow, the API endpoints, and `cao flow run` updated to await. - memory_service curated path and the flow CLI fixed to the new contract. Full unit suite green (1908 passed); black + isort clean.
Summary
CAO_EAGER_INBOX_DELIVERYenv var) that delivers inbox messages to terminals in PROCESSING or WAITING_USER_ANSWER states, eliminatinginter-turn latency for capable providers
_initializedflag prevents premature delivery during terminal startup--agent <name>for Claude Code's native agent store (~/.claude/agents/)Misc Claude Code fixes:
CLAUDE_CODE_EFFORT_LEVELthrough CAO's env-unset command>in content (Java generics, git diffs, HTML) using start-of-line anchorTest plan