Summary

The verdict parsing regex in ai-review action fails to extract verdicts when the AI includes brackets around the verdict token (e.g., VERDICT: [PASS] instead of VERDICT: PASS).

Root Cause Analysis

The Problem

1. Prompt templates show options with brackets: Files like spec-trace-requirements.md (line 54) and spec-check-completeness.md (line 60) show:

   VERDICT: [PASS|PARTIAL|FAIL]
   

2. AI interprets brackets literally: Instead of outputting VERDICT: PASS, the AI outputs VERDICT: [PASS] including the brackets.

3. Regex can't match brackets: The verdict parsing in action.yml line 705:

   verdict=$(echo "$output" | sed -n 's/.*VERDICT:[[:space:]]*\([A-Z_]*\).*/\1/p' | tail -n 1)

   This regex [A-Z_]* cannot match [ so when parsing VERDICT: [PASS]:

   • After VERDICT: there's a space (matched by [[:space:]]*)
   • The capture group tries to match from [ but [ is not in [A-Z_]
   • Capture group is empty
   • Falls through to NEEDS_REVIEW (line 721)

Evidence

From PR #543 CI run 20588542269:

• AI Output: VERDICT: [PASS] with requirements traceability showing 100% coverage
• Parsed verdict: TRACE_VERDICT: NEEDS_REVIEW
• Result: Job failed despite all requirements being satisfied

Proposed Fixes

Option A: Fix the regex (Quick fix)

Update line 705 to handle optional brackets:

verdict=$(echo "$output" | sed -n 's/.*VERDICT:[[:space:]]*\[\?\([A-Z_]*\)\]\?.*/\1/p' | tail -n 1)

Option B: Fix the prompts (Root cause)

Update prompt templates to show expected output without the OR-style bracket notation:

# Before (confusing)
VERDICT: [PASS|PARTIAL|FAIL]

# After (explicit)
**VERDICT**: PASS, PARTIAL, or FAIL (use exact keyword, no brackets)

Example: VERDICT: PASS

Option C: Both (Recommended)

1. Fix regex for defense-in-depth
2. Update prompts for clarity

Affected Files

• .github/actions/ai-review/action.yml (parsing)
• .github/prompts/spec-trace-requirements.md (prompt template)
• .github/prompts/spec-check-completeness.md (prompt template)
• Potentially other prompts using [OPTION1|OPTION2] notation

Impact

• PRs with PASS verdicts incorrectly fail
• Requires manual investigation to determine actual status
• False negatives reduce trust in CI