Skip to content

fix: detect active permission prompts using line-based counting#71

Merged
haofeif merged 1 commit into
awslabs:mainfrom
sriharshaarangi:fix/permission-prompt-detection
Feb 16, 2026
Merged

fix: detect active permission prompts using line-based counting#71
haofeif merged 1 commit into
awslabs:mainfrom
sriharshaarangi:fix/permission-prompt-detection

Conversation

@sriharshaarangi

Copy link
Copy Markdown
Contributor

Problem

Permission prompt detection in get_status() fails in two ways depending on the approach:

  1. PR fix: prevent permission prompt pattern from matching stale prompts #69 (current main): Pattern [ \t]* between [y/n/t]: and idle prompt can't bridge newlines. Active permission prompts are never detected — get_status() returns IDLE instead of WAITING_USER_ANSWER. Messages delivered during active prompts corrupt the terminal.

  2. PR fix: Handle CLI prompts with trailing text #61 (previous approach): Pattern \s* bridges newlines but also matches stale (already-answered) prompts. After the user answers y, the old [y/n/t]: text remains in the terminal buffer. get_status() keeps returning WAITING_USER_ANSWER even after the command completes.

Both approaches fail because they try to combine the permission pattern and idle pattern into a single regex, which can't distinguish active from stale prompts.

Solution

Decouple permission detection from idle detection using line-based counting:

  1. Match [y/n/t]: independently (non-greedy, no idle pattern suffix)
  2. Find the last match (handles re-rendered prompts)
  3. Count lines containing the idle prompt after the last [y/n/t]:
  4. ≤1 idle line = active prompt → WAITING_USER_ANSWER
  5. 1 idle lines = stale prompt → fall through to other status checks

This works because:

  • Active prompt: CLI renders [y/n/t]: then at most one idle prompt on the next line
  • Stale prompt: user answered, agent produced output, multiple idle prompts follow
  • \r redraws stay on the same line (no \n), so they don't inflate the count

Applied to both kiro_cli.py and q_cli.py.

Test Results

Unit Tests (24 tests — test_permission_prompt_detection.py)

Category Tests Our Fix #61 #69
Active prompts (P1-P4, P8) 5 ✅ 5/5 ✅ 5/5 ❌ 0/5
Stale prompts (P5-P7) 4 ✅ 4/4 ❌ 0/4 ✅ 4/4*
Non-permission (N1-N9) 8 ✅ 8/8 ✅ 8/8 ✅ 8/8
Edge cases (ANSI, multi-prompt) 7 ✅ 7/7 ❌ 4/7 ❌ 4/7
Total 24 24/24 18/24 15/24

* #69 stale tests pass but only because permission prompts are never detected (right result, wrong reason).

Integration Tests (7 tests — test_kiro_cli_integration.py, real kiro-cli)

Test Our Fix #61 #69
P1/P2 active prompt ❌ returns IDLE
P3/P4 injection during prompt ❌ returns IDLE
P5/P6 stale after answer ❌ returns WAITING_USER_ANSWER ✅*
P7 second prompt after first answered ❌ first command never completes ❌ second prompt not detected
N4/N5 processing state
INIT (smoke)
QUERY (smoke)
Total 7/7 5/7 4/7

* #69 P5/P6 passes for wrong reason (nothing ever matches).

Integration tests support CAO_TEST_WATCH=1 to open a Terminal.app window attached to the tmux session for live debugging, and auto-dump terminal output on failure.

Files Changed

  • src/cli_agent_orchestrator/providers/kiro_cli.py — line-based counting in get_status()
  • src/cli_agent_orchestrator/providers/q_cli.py — same fix
  • test/providers/test_permission_prompt_detection.py — 24 unit tests with 6 fixture files
  • test/providers/test_kiro_cli_integration.py — 7 integration tests against real kiro-cli
  • test/providers/fixtures/ — 6 fixture files with real ANSI terminal captures

@codecov-commenter

codecov-commenter commented Feb 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@a70a43f). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #71   +/-   ##
=======================================
  Coverage        ?   20.97%           
=======================================
  Files           ?       30           
  Lines           ?     1392           
  Branches        ?        0           
=======================================
  Hits            ?      292           
  Misses          ?     1100           
  Partials        ?        0           
Flag Coverage Δ
unittests 20.97% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sriharshaarangi sriharshaarangi force-pushed the fix/permission-prompt-detection branch from 5f70dfc to 3835385 Compare February 10, 2026 07:28
Decouple permission pattern from idle pattern in get_status().
Count idle prompt lines after last [y/n/t]: to distinguish active
(<=1 line) from stale (>1 lines) permission prompts.

Previous approaches failed because:
- awslabs#69: [ \t]* can't bridge newlines, active prompts never detected
- awslabs#61: \s* bridges newlines but also matches stale prompts

Line-based counting handles \r redraws correctly (same line, no \n)
and uses the LAST [y/n/t]: match to handle re-rendered prompts.

Applied to both kiro_cli.py and q_cli.py.

Unit tests (24 tests, test_permission_prompt_detection.py):

  Category                  | Our Fix | awslabs#61    | awslabs#69
  Active prompts (P1-P4,P8) | 5/5     | 5/5    | 0/5
  Stale prompts (P5-P7)     | 4/4     | 0/4    | 4/4*
  Non-permission (N1-N9)    | 8/8     | 8/8    | 8/8
  Edge cases (ANSI, multi)  | 7/7     | 4/7    | 4/7
  Total                     | 24/24   | 18/24  | 15/24

  * awslabs#69 stale tests pass only because prompts are never detected.

Integration tests (7 tests, test_kiro_cli_integration.py, real kiro-cli):

  Test                | Our Fix | awslabs#61                        | awslabs#69
  P1/P2 active        | PASS    | PASS                       | FAIL (IDLE)
  P3/P4 injection     | PASS    | PASS                       | FAIL (IDLE)
  P5/P6 stale         | PASS    | FAIL (WAITING_USER_ANSWER) | PASS*
  P7 multiple prompts | PASS    | FAIL (never completes)     | FAIL (not detected)
  N4/N5 processing    | PASS    | PASS                       | PASS
  INIT smoke          | PASS    | PASS                       | PASS
  QUERY smoke         | PASS    | PASS                       | PASS
  Total               | 7/7     | 5/7                        | 4/7

  * awslabs#69 P5/P6 passes for wrong reason (nothing ever matches).

Opt-in live terminal view via CAO_TEST_WATCH=1.
@sriharshaarangi sriharshaarangi force-pushed the fix/permission-prompt-detection branch from 3835385 to 79a267f Compare February 10, 2026 07:32
@haofeif haofeif requested a review from a team February 10, 2026 12:02
@haofeif haofeif added the bug Something isn't working label Feb 10, 2026

@anilkmr-a2z anilkmr-a2z left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this.

@haofeif haofeif merged commit 5cb55d4 into awslabs:main Feb 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants