Skip to content

feat(agents): implement Phase 2 consistency fixes for cross-document validation#49

Merged
rjmurillo merged 4 commits into
mainfrom
copilot/remediate-coderabbit-pr-43
Dec 16, 2025
Merged

feat(agents): implement Phase 2 consistency fixes for cross-document validation#49
rjmurillo merged 4 commits into
mainfrom
copilot/remediate-coderabbit-pr-43

Conversation

Copilot AI commented Dec 16, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

✅ Phase 2 Complete: All 4 consistency fixes (P1) implemented with proper workflow

Changes

Workflow Correction

  • Updated shared templates in templates/agents/ (the source of truth)
  • Ran Generate-Agents.ps1 to regenerate platform-specific agents
  • Created retrospective and handoff documentation

P1-1: Critic Escalation Template

  • Template: templates/agents/critic.shared.md
  • Generated: src/copilot-cli/critic.agent.md, src/vs-code-agents/critic.agent.md
  • Added "Escalation Prompt Completeness Requirements" with Verified Facts table
  • Added anti-pattern: "Converting exact values to ranges loses actionable detail"

P1-2: Task Generator Estimate Reconciliation

  • Template: templates/agents/task-generator.shared.md
  • Generated: src/copilot-cli/task-generator.agent.md, src/vs-code-agents/task-generator.agent.md
  • Added 10% threshold rule for estimate divergence
  • Added reconciliation process and output template

P1-3: Planner Condition Traceability

  • Template: templates/agents/planner.shared.md
  • Generated: src/copilot-cli/planner.agent.md, src/vs-code-agents/planner.agent.md
  • Added Work Breakdown template with Conditions column
  • Added validation checklist for orphan conditions

P1-4: Cross-Document Validation CI

  • Created build/scripts/Validate-PlanningArtifacts.ps1
  • Created build/scripts/tests/Validate-PlanningArtifacts.Tests.ps1 (17 tests passing)
  • NEW: Added .github/workflows/validate-planning-artifacts.yml for CI
  • NEW: Added planning artifacts validation to .githooks/pre-commit

Handoff Artifacts

  • .agents/retrospective/phase2-workflow-learnings.md - Workflow learnings
  • .agents/planning/phase2-complete-handoff.md - Handoff for future agents
  • .agents/skills/agent-generation-workflow.md - Skill entry for agent workflow

Verification

# Validate agents match templates
pwsh build/Generate-Agents.ps1 -Validate  # ✅ PASSED

# Run planning artifact validation
pwsh build/scripts/Validate-PlanningArtifacts.ps1  # ✅ PASSED

# Run Pester tests
Invoke-Pester build/scripts/tests/Validate-PlanningArtifacts.Tests.ps1  # ✅ 17/17 PASSED

Checklist

  • P1-1: Update critic with escalation template
  • P1-2: Update task-generator with estimate reconciliation
  • P1-3: Update planner with condition traceability
  • P1-4: Create cross-document validation CI
  • Update shared templates (templates/agents/*.shared.md)
  • Run Generate-Agents.ps1 to regenerate agents
  • Validate all generated agents match templates
  • Create retrospective with workflow learnings
  • Create handoff artifacts for future agents
  • Add skill entry for agent generation workflow
  • Add GitHub workflow for planning artifacts validation
  • Add planning artifacts validation to pre-commit hook
Original prompt

This section details on the original issue you should resolve

<issue_title>Agent Quality: Remediate CodeRabbit PR #43 Findings (7 Issues, 15 Agent Gaps)</issue_title>
<issue_description>## Summary

CodeRabbit review of PR #43 (feat/templates) identified 7 issues stemming from 5 systemic patterns and 15 agent capability gaps across 9 agents. This issue tracks remediation across 4 workstreams with ~17.5 hours total effort.

Root Cause Analysis

ID Issue Severity Pattern Responsible Agent(s)
I1 Escalation prompt missing critical data Major Information Loss critic
I2 QA conditions not tracked in work breakdown Major Cross-Document Consistency planner, orchestrator
I3 Absolute Windows paths in references Critical Environment Contamination explainer
I4 Effort estimate discrepancy (12-16 vs 8-14 hrs) Major Cross-Document Consistency task-generator
I5 Naming convention violation (no PREFIX-NNN) Major Naming Drift roadmap
I6 Memory estimate inconsistency Major Cross-Document Consistency memory
I7 Security analysis incomplete (post-impl) Critical Single-Phase Security security, implementer

Systemic Patterns Identified

  1. Cross-Document Consistency - No automated validation between epic, tasks, PRD, memory
  2. Information Loss During Synthesis - Data summarized away during document generation
  3. Environment Contamination - Absolute paths leak into documentation
  4. Naming Convention Drift - Inconsistent artifact naming
  5. Single-Phase Security Review - No post-implementation verification

Phase 1: Critical Fixes (P0) - Immediate

Effort: 5 hours

  • P0-1 Update src/claude/explainer.md with path normalization requirements
    • Add validation regex: [A-Z]:\|\/Users\/|\/home\/
    • Include anti-pattern example
  • P0-2 Update src/claude/security.md with post-implementation verification
    • Add security-relevant change triggers
    • Add PIV (Post-Implementation Verification) template
  • P0-3 Update src/claude/implementer.md with security flagging protocol
    • Add self-assessment triggers table
    • Add handoff note requirement for security-relevant code
  • P0-4 Create path normalization CI
    • Script: build/Validate-PathNormalization.ps1
    • Workflow: .github/workflows/validate-paths.yml

Phase 2: Consistency Fixes (P1) - Next PR

Effort: 6.5 hours

  • P1-1 Update src/claude/critic.md with escalation template
    • Add mandatory escalation data template with Verified Facts table
    • Add anti-pattern: "Converting exact values to ranges"
  • P1-2 Update src/claude/task-generator.md with estimate reconciliation
    • Add 10% threshold for flagging discrepancies
    • Add reconciliation protocol and output template
  • P1-3 Update src/claude/planner.md with condition traceability
    • Add Work Breakdown template with Conditions column
    • Add validation checklist for orphan conditions
  • P1-4 Create cross-document validation CI
    • Script: build/Validate-PlanningArtifacts.ps1
    • Validates estimate consistency (20% threshold)
    • Validates condition-to-task traceability

Phase 3: Process Improvements (P2) - Within 2 Weeks

Effort: 4 hours

  • P2-1 Update src/claude/roadmap.md with naming conventions
    • Add EPIC-NNN-[name].md pattern
    • Add numbering rules
  • P2-2 Update src/claude/memory.md with freshness protocol
    • Add update triggers when downstream refinements occur
    • Add source tracking in observations
  • P2-3 Update src/claude/orchestrator.md with consistency checkpoint
    • Add pre-critic validation checkpoint
    • Add failure action (return to planner)
  • P2-4 Create .agents/governance/naming-conventions.md
    • Sequenced artifact patterns (EPIC-NNN, ADR-NNN, TM-NNN)
    • Type-prefixed patterns (prd-, tasks-, etc.)
  • P2-5 Create .agents/governance/consistency-protocol.md
    • Checkpoint locations (after task-generator, after implementation)
    • Inconsistency response procedure

Phase 4: Polish (P3) - As Capacity Allows

Effort: 2 hours

  • P3-1 Add handoff validation to all agents (critic, implementer, qa, task-generator)
  • P3-2 Update CLAUDE.md with naming reference

Skills to Extract (After Completion)

Skill Statement Evidence
Skill-Review-001 Include all verified facts with exact values in escalation prompts Issue 1 - lost 99%/60-70% breakdown
Skill-Doc-002 Convert absolute paths to relative before committing Issue 3 - Windows paths in References
Skill-Plan-003 Derived estimates differing >10% from source require reconciliation Issues 4, 6 - 43% difference
Skill-Security-001 Security-relevant implementations require post-implementation verific...

Custom agent used: orchestrator
Autonomous task orchestrator that coordinates specialized agents end-to-end. Routes work to appropriate agents, manages handoffs, and ensures complete task execution. Use for complex multi-step tasks requiring multiple agent specializations or when task routing is unclear.

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Note

Implements Phase 2 consistency fixes by updating shared agent templates and regenerating platform agents, adds a planning-artifact validation script with tests, and includes handoff/retrospective documentation.

  • Agents/Templates
    • Critic (templates/agents/critic.shared.md): Add Escalation Prompt Completeness Requirements with Verified Facts table and anti-pattern; propagate to src/copilot-cli/critic.agent.md, src/vs-code-agents/critic.agent.md.
    • Task Generator (templates/agents/task-generator.shared.md): Add 10% estimate divergence rule, reconciliation process and output template; update handoff steps; propagate to src/copilot-cli/task-generator.agent.md, src/vs-code-agents/task-generator.agent.md.
    • Planner (templates/agents/planner.shared.md): Add Condition-to-Task Traceability with Work Breakdown "Conditions" column, checklist, and anti-pattern; propagate to src/copilot-cli/planner.agent.md, src/vs-code-agents/planner.agent.md.
  • Validation/CI
    • Add build/scripts/Validate-PlanningArtifacts.ps1 to check estimate consistency, orphan conditions, and document structure; configurable thresholds/exit behavior.
    • Add tests build/scripts/tests/Validate-PlanningArtifacts.Tests.ps1 (17 tests).
  • Docs/Handoff
    • Add handoff .agents/planning/phase2-complete-handoff.md.
    • Add retrospective .agents/retrospective/phase2-workflow-learnings.md.
    • Add skill .agents/skills/agent-generation-workflow.md (workflow: templates → generate → verify).

Written by Cursor Bugbot for commit a17c39c. This will update automatically on new commits. Configure here.

@rjmurillo
feat(agents): implement Phase 2 consistency fixes (P1-1 through P1-4)
62883bd 
Co-authored-by: rjmurillo <6811113+rjmurillo@users.noreply.github.com>
Copilot AI changed the title [WIP] Remediate CodeRabbit PR #43 findings feat(agents): implement Phase 2 consistency fixes for cross-document validation Dec 16, 2025
Copilot AI requested a review from rjmurillo December 16, 2025 21:44
rjmurillo
rjmurillo requested changes Dec 16, 2025
View reviewed changes
Comment thread src/claude/critic.md
@rjmurillo
fix(agents): sync shared templates and regenerate platform agents
a17c39c 
Co-authored-by: rjmurillo <6811113+rjmurillo@users.noreply.github.com>
Copilot AI requested a review from rjmurillo December 16, 2025 23:02
rjmurillo
rjmurillo reviewed Dec 16, 2025
View reviewed changes
Comment thread build/scripts/Validate-PlanningArtifacts.ps1
@rjmurillo rjmurillo marked this pull request as ready for review December 16, 2025 23:09
Copilot AI review requested due to automatic review settings December 16, 2025 23:09
@rjmurillo
feat(ci): add planning artifacts validation workflow and pre-commit hook
fe7962b 
Co-authored-by: rjmurillo <6811113+rjmurillo@users.noreply.github.com>
Copilot AI reviewed Dec 16, 2025
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements Phase 2 consistency fixes for cross-document validation across agent planning artifacts. The changes address systemic issues identified in a previous CodeRabbit review by adding validation protocols to prevent estimate divergence and orphan conditions. Notably, the PR also corrects a workflow issue where changes were initially made only to Claude agents without updating the shared templates, demonstrating the importance of following the established agent generation workflow.

Key Changes

  • Added Estimate Reconciliation Protocol to task-generator agent with 10% threshold rule for detecting estimate drift between epics/PRDs and task breakdowns
  • Added Escalation Prompt Completeness Requirements to critic agent ensuring verified facts are preserved with exact values during escalations
  • Added Condition-to-Task Traceability to planner agent preventing orphan conditions by requiring task linkage in work breakdown tables

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated no comments.

Show a summary per file
File Description
templates/agents/task-generator.shared.md Added estimate reconciliation protocol with 10% threshold rule and reconciliation process
templates/agents/critic.shared.md Added escalation prompt requirements with verified facts table template
templates/agents/planner.shared.md Added condition traceability requirements with work breakdown template
src/vs-code-agents/task-generator.agent.md Generated agent file matching task-generator template changes
src/vs-code-agents/critic.agent.md Generated agent file matching critic template changes
src/vs-code-agents/planner.agent.md Generated agent file matching planner template changes
src/copilot-cli/task-generator.agent.md Generated agent file matching task-generator template changes
src/copilot-cli/critic.agent.md Generated agent file matching critic template changes
src/copilot-cli/planner.agent.md Generated agent file matching planner template changes
src/claude/task-generator.md Manually maintained Claude agent with estimate reconciliation protocol
src/claude/critic.md Manually maintained Claude agent with escalation requirements
src/claude/planner.md Manually maintained Claude agent with condition traceability
build/scripts/Validate-PlanningArtifacts.ps1 New validation script for cross-document consistency (estimate divergence, orphan conditions)
build/scripts/tests/Validate-PlanningArtifacts.Tests.ps1 Comprehensive Pester test suite with 17 tests covering validation scenarios
.agents/skills/agent-generation-workflow.md Skill documentation capturing the agent generation workflow pattern
.agents/retrospective/phase2-workflow-learnings.md Retrospective documenting workflow learnings from Phase 2 implementation
.agents/planning/phase2-complete-handoff.md Handoff documentation for future agents continuing this work

Copilot AI requested a review from rjmurillo December 16, 2025 23:13
@rjmurillo rjmurillo added the enhancement New feature or request label Dec 16, 2025
@rjmurillo rjmurillo merged commit 48978ce into main Dec 16, 2025
12 of 14 checks passed
@rjmurillo rjmurillo deleted the copilot/remediate-coderabbit-pr-43 branch December 16, 2025 23:18
@github-actions github-actions Bot mentioned this pull request Dec 20, 2025
Merged
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@coderabbitai coderabbitai[bot] coderabbitai[bot] approved these changes

+1 more reviewer

@rjmurillo rjmurillo rjmurillo approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@rjmurillo rjmurillo

Copilot code review Copilot

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Agent Quality: Remediate CodeRabbit PR #43 Findings (7 Issues, 15 Agent Gaps)

3 participants

@rjmurillo