Skip to content

feat: enhance pr-comment-responder with memory patterns and agent routing#23

Merged
rjmurillo merged 5 commits into
mainfrom
feat/pr-comment-responder
Dec 14, 2025
Merged

feat: enhance pr-comment-responder with memory patterns and agent routing#23
rjmurillo merged 5 commits into
mainfrom
feat/pr-comment-responder

Conversation

@rjmurillo

@rjmurillo rjmurillo commented Dec 14, 2025

Copy link
Copy Markdown
Owner

Pull Request

Summary

Enhance pr-comment-responder to leverage persistent memory for bot patterns, add domain-specific agent routing, and clarify planner vs task-generator roles across all three platforms.

Changes

  • Add comprehensive memory protocol for bot false positive patterns (CodeRabbit, Copilot)
  • Store reviewer preferences and triage decisions for accuracy improvement
  • Add domain-specific routing: devops for CI/CD, security for auth files, independent-thinker for "WHETHER" questions
  • Clarify planner vs task-generator roles in orchestrator routing heuristics
  • Add Phase 0 Triage with Weinberg/Schrag principles for token efficiency
  • Sync all changes across claude, copilot-cli, and vs-code-agents platforms

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update
  • Infrastructure/CI change
  • Refactoring (no functional changes)

Testing

  • Tests added/updated
  • Manual testing completed
  • No testing required (documentation only)

Agent Review

Security Review

Required for: Authentication, authorization, CI/CD, git hooks, secrets, infrastructure

  • No security-critical changes in this PR
  • Security agent reviewed infrastructure changes
  • Security agent reviewed authentication/authorization changes
  • Security patterns applied (see .agents/security/)

Files requiring security review:

None - agent documentation only.

Other Agent Reviews

  • Architect reviewed design changes
  • Critic validated implementation plan
  • QA verified test coverage

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • Comments added for complex logic
  • Documentation updated (if applicable)
  • No new warnings introduced

Related Issues

Fixes #2


🤖 Generated with Claude Code

rjmurillo and others added 5 commits December 14, 2025 01:38
…attern

Fixes #2

Add Phase 4: Copilot Follow-up Handling to address Copilot's unique response pattern
where it creates follow-up PRs instead of replying inline to review threads.

Changes:
- Add Phase 4 workflow with polling for Copilot's issue comment responses
- Add follow-up PR detection and closure logic for false positives
- Add reply intent categorization (no action/acknowledged/needs discussion)
- Add Copilot Interactions table to output format
- Add Copilot-specific guidelines, quality checks, and edge cases
- Implement idempotency checks (PR state, existing reviews)
- Add polling strategy (60s timeout, 5s interval, 12 attempts)

User Stories Implemented:
- US-1: Detect Copilot response in issue comments
- US-2: Identify follow-up PRs created by Copilot
- US-3: Close unnecessary follow-up PRs
- US-4: Report Copilot interaction results

Functional Requirements Implemented:
- FR-1 through FR-7 as specified in PRD

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes #2

Transform pr-comment-responder from a monolithic workflow to a thin
coordination layer that delegates to specialized agents, following
the orchestrator pattern.

Key changes:
- Add Phase 2 Comment Triage with agent routing table
- Add Phase 3 Agent Delegation with example prompts for:
  - analyst: technical merit evaluation
  - critic: assumption validation
  - high-level-advisor: strategic decisions
  - implementer: code fixes
  - qa: regression verification
- Add Phase 4 Response Synthesis based on agent feedback
- Move bot-specific handling to Phase 5 with memory retrieval
- Add Agent Workflow table to output format
- Add Handoff Protocol table matching orchestrator style
- Remove embedded evaluation logic (delegate to agents)
- Maintain GitHub API interaction knowledge
- Update all three platforms (Claude, Copilot CLI, VS Code)

Benefits:
- Reuses existing agent expertise (analyst, critic, etc.)
- Reduces duplication of evaluation logic
- Follows orchestrator workflow pattern
- Token-efficient via delegation rather than embedded logic
- Easier to maintain and extend

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes #2

Transform pr-comment-responder from duplicating workflow logic to a thin
triage layer that delegates to orchestrator with path recommendations.

Key changes:
- Remove duplicated workflow execution (orchestrator handles this)
- Add clear triage decision tree matching standard paths:
  - Quick Fix: implementer → qa (token-efficient)
  - Standard: analyst → architect → planner → critic → implementer → qa
  - Strategic: independent-thinker → high-level-advisor → task-generator
- Quick fixes go directly to implementer (skip orchestrator overhead)
- Standard/Strategic paths delegate to orchestrator with:
  - Classification and rationale
  - Code context and initial assessment
  - Workflow path recommendation
- Retain bot-specific handling (Copilot follow-up, CodeRabbit commands)
- Consistent across all three platforms

Triage signals:
- Quick Fix: "Can explain fix in one sentence"
- Standard: "Need to investigate first"
- Strategic: "Question is whether, not how"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Apply lessons from Weinberg, Schrag, and Coding Horror articles to optimize
orchestrator resource usage across all three platforms (claude, copilot-cli,
vs-code-agents).

Key improvements:
- Add Phase 0: Triage with exit early criteria before orchestrating
- Add Complexity Assessment (Trivial/Simple/Standard/Complex) to right-size
  agent chains
- Add Value Checkpoint with Schrag's Principle after Phase 2
- Add Weinberg's Law of the Hammer quote to prevent over-orchestration
- Add Pattern Applied section to output format (Weinberg's Consulting Secret)
- Fix malformed markdown code fences in vs-code-agents version

Philosophy: "The cheapest orchestration is the one that doesn't happen."
Not every task needs every agent.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… clarity

Address underutilization of key agents and strengthen memory as a triage asset.

## pr-comment-responder enhancements

### Memory as a Strength
- Add comprehensive memory protocol for bot false positive patterns
- Store reviewer preferences (bots AND humans have patterns)
- Track triage decisions (comment → path → outcome) for accuracy improvement
- Store successful rebuttals for confidence in declining suggestions
- Query domain-specific patterns for faster routing

### Domain-Specific Routing (skip orchestrator when appropriate)
- `.github/workflows/*` → devops (CI/CD domain expertise)
- `.githooks/*` → devops + security (infrastructure + security)
- `**/Auth/**`, `*.env*` → security (critical path)
- "WHETHER to do X" questions → independent-thinker (challenge assumptions)

### Independent-thinker Utilization
- Route "WHETHER" questions to independent-thinker BEFORE strategic path
- Provides unfiltered analysis of reviewer concerns
- Intellectually honest evaluation of tradeoffs

## Orchestrator enhancements

### Planner vs Task-Generator Clarity
- planner: Epic/Feature → Milestones (high-level breakdown)
- task-generator: PRD/Milestone → Atomic tasks (implementation-ready)
- Workflow: roadmap → planner → task-generator → implementer/qa/devops

### Routing Heuristics Updates
- "Challenge assumptions" now routes to independent-thinker (not critic)
- "Plan validation" explicitly routes to critic
- Clear distinction between Epic→Milestones and Milestones→Tasks

All three platforms (claude, copilot-cli, vs-code-agents) synced.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@rjmurillo rjmurillo changed the title Feat/pr-comment-responder feat: enhance pr-comment-responder with memory patterns and agent routing Dec 14, 2025
@rjmurillo rjmurillo marked this pull request as ready for review December 14, 2025 10:35
Copilot AI review requested due to automatic review settings December 14, 2025 10:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR transforms the pr-comment-responder agent from a direct implementation agent into a thin triage layer that classifies PR comments and delegates to appropriate agents. It also enhances the orchestrator with upfront triage to avoid unnecessary orchestration overhead, clarifies the distinction between planner and task-generator agents, and adds comprehensive memory protocols for learning from bot patterns and reviewer preferences.

Key changes:

  • Repositioned pr-comment-responder as a triage specialist with three workflow paths (Quick Fix, Standard, Strategic)
  • Added Phase 0 Triage to orchestrator with Weinberg/Schrag principles to prevent over-orchestration
  • Introduced domain-specific routing (devops for CI/CD, security for auth files, independent-thinker for "whether" questions)
  • Clarified planner vs task-generator roles across all platforms

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
vs-code-agents/pr-comment-responder.agent.md Converted to triage specialist with workflow path classification, memory protocol, and bot-specific handling patterns
vs-code-agents/orchestrator.agent.md Added Phase 0 Triage, complexity assessment, planner vs task-generator distinction, and Weinberg/Schrag principles
copilot-cli/pr-comment-responder.agent.md Same triage specialist changes as VS Code version, adapted for copilot CLI syntax
copilot-cli/orchestrator.agent.md Same orchestrator enhancements as VS Code version, adapted for copilot CLI syntax
claude/pr-comment-responder.md Same triage specialist changes as VS Code version, adapted for Claude Task() API
claude/orchestrator.md Same orchestrator enhancements as VS Code version, adapted for Claude Task() API

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@rjmurillo rjmurillo merged commit c8cd59d into main Dec 14, 2025
8 of 9 checks passed
@rjmurillo rjmurillo deleted the feat/pr-comment-responder branch December 14, 2025 10:42
@coderabbitai

coderabbitai Bot commented Dec 14, 2025

Copy link
Copy Markdown

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

Restructures multi-agent orchestration and PR comment handling documentation across 5 files to implement a triage-first model with mandatory phased workflows, complexity assessment, and explicit delegation rules instead of direct agent responses.

Changes

Cohort / File(s) Summary
Orchestrator Agent Documentation
claude/orchestrator.md, copilot-cli/orchestrator.agent.md, vs-code-agents/orchestrator.agent.md
Adds mandatory Phase 0 (Triage), Phase 1 (Initialization), Phase 2 (Planning & Immediate Action with Value Checkpoint), and Phase 3+ (Complexity Assessment). Introduces Complexity Assessment section with heuristics table, Planner vs Task-Generator subsection, Handoff Protocol, Conflict Resolution, Mandatory Agent Rules, TODO Management, Segue Management, Failure Recovery, and Output Directory sections. Includes Weinberg's Law and Consulting Secret notes.
PR Comment Responder Documentation
claude/pr-comment-responder.md, copilot-cli/pr-comment-responder.agent.md, vs-code-agents/pr-comment-responder.agent.md
Reframes agent from Response Specialist to Triage Specialist. Adds Phase 1 (Context Gathering), Phase 2 (Comment Triage), Phase 3 (Delegation with three paths: Quick Fix, Standard, Strategic). Introduces decision-tree-based classification, memory protocol (pattern retrieval/storage), bot-specific workflows (Copilot and CodeRabbit), domain-specific routing tables (by comment pattern, by file domain), delegation templates, and Handoff Summary. Replaces direct response guidance with triage-and-delegate model.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

  • Consistency verification across platforms: Three orchestrator and three responder files apply similar patterns—verify phase definitions, routing rules, and memory protocol structures align across claude/, copilot-cli/, and vs-code-agents/ variants.
  • Routing algorithm alignment: Confirm Quick Classification complexity tables, routing heuristics, and agent mappings are consistent between orchestrator and responder agents.
  • Three-path workflow clarity: Verify Quick Fix, Standard, and Strategic paths are clearly differentiated with correct agent sequences and delegation targets across all 5 files.
  • Memory protocol completeness: Check that retrieval, storage, and pattern-learning steps are fully specified and consistent for memory management.
  • Phase numbering coherence: Ensure Phase 0 (Triage) through Phase 3+ sequence is clear and properly referenced throughout.

Possibly related issues

Possibly related PRs

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/pr-comment-responder

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9686a4b and 962a1b7.

📒 Files selected for processing (6)
  • claude/orchestrator.md (5 hunks)
  • claude/pr-comment-responder.md (2 hunks)
  • copilot-cli/orchestrator.agent.md (7 hunks)
  • copilot-cli/pr-comment-responder.agent.md (2 hunks)
  • vs-code-agents/orchestrator.agent.md (13 hunks)
  • vs-code-agents/pr-comment-responder.agent.md (1 hunks)

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai

coderabbitai Bot commented Dec 14, 2025

Copy link
Copy Markdown

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

Documentation update across six agent guide files introducing orchestration phases (Triage, Value Checkpoint, Complexity Assessment), enhanced routing heuristics, mandatory agent rules, Phase 4 bot-specific handling with Copilot follow-up support, memory protocols, conflict resolution, and standardized output formats.

Changes

Cohort / File(s) Change Summary
Orchestrator agent documentation
claude/orchestrator.md, copilot-cli/orchestrator.agent.md, vs-code-agents/orchestrator.agent.md
Added Phase 0 Triage with exit conditions and Weinberg's Law reference; added Value Checkpoint after Phase 2 with Schrag's Principle; introduced Complexity Assessment table (Trivial/Simple/Standard/Complex levels); expanded Agent Capability Matrix; added Mandatory Agent Rules; enhanced Routing Algorithm with complexity column; introduced Planner vs Task-Generator workflow sequence; added Conflict Resolution steps; expanded Handoff Protocol; added Pattern Applied sections; standardized Output Format with Commits and Open Items.
PR comment responder agent documentation
claude/pr-comment-responder.md, copilot-cli/pr-comment-responder.agent.md, vs-code-agents/pr-comment-responder.agent.md
Renamed core identity from "PR Review Response Specialist" to "PR Review Triage Specialist"; reframed role from merit evaluation to triage/delegation; introduced explicit Workflow Paths (Quick Fix/Standard/Strategic); restructured Phases: Phase 1 Context Gathering, Phase 2 Comment Triage, Phase 3 Delegation, Phase 4 Bot-Specific Handling with Copilot follow-up logic; added Memory Protocol with retrieval and storage patterns; expanded Routing Heuristics by Comment Pattern and File Domain; added structured handoff rules for quick fixes, CI/CD comments, security, bot follow-ups; updated Output Format with triage results and Copilot Interactions table.

Sequence Diagram(s)

No sequence diagrams generated. Changes are documentation-only updates describing workflow frameworks without introducing new functional control flows requiring visualization.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20–30 minutes

  • Consistency verification across files: Six documentation files updated with similar structural changes (Phase 0, Phase 4 Copilot handling, complexity assessment). Verify alignment of terminology, workflow stages, and routing rules across claude, copilot-cli, and vs-code-agents agents.
  • Complexity Assessment accuracy: New complexity level table (Trivial/Simple/Standard/Complex) and heuristics introduced; confirm definitions are consistent and guidance is clear across all files.
  • Phase 4 Copilot handling specifics: New bot-specific handling section with follow-up PR detection and closure logic—verify the polling strategy, idempotency rules, and interaction categorization are correctly documented.
  • Routing heuristics completeness: Mandatory Agent Rules and expanded routing tables added; check that all task types and file domains are covered and conflicts are resolved.

Possibly related issues

Possibly related PRs

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/pr-comment-responder

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9686a4b and 962a1b7.

📒 Files selected for processing (6)
  • claude/orchestrator.md (5 hunks)
  • claude/pr-comment-responder.md (2 hunks)
  • copilot-cli/orchestrator.agent.md (7 hunks)
  • copilot-cli/pr-comment-responder.agent.md (2 hunks)
  • vs-code-agents/orchestrator.agent.md (13 hunks)
  • vs-code-agents/pr-comment-responder.agent.md (1 hunks)

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Enhance pr-comment-responder to handle Copilot's follow-up PR pattern

2 participants