Skip to content

Add Multi-Agent Impact Analysis Framework for planning phase#40

Merged
rjmurillo merged 9 commits into
mainfrom
copilot/add-impact-analysis-framework
Dec 15, 2025
Merged

Add Multi-Agent Impact Analysis Framework for planning phase#40
rjmurillo merged 9 commits into
mainfrom
copilot/add-impact-analysis-framework

Conversation

Copilot AI commented Dec 15, 2025

Copy link
Copy Markdown
Contributor

Multi-Agent Impact Analysis Framework Implementation ✅

All Phases Complete

  • Phase 1: Core Framework Design

    • Document the Agent Consultation Protocol in planner agents
    • Define specialist agent roles and impact analysis focus areas
    • Create impact analysis prompt templates for each specialist
  • Phase 2: Update Planner Agent Instructions

    • Add consultation orchestration guidance to claude/planner.md
    • Add consultation orchestration guidance to copilot-cli/planner.agent.md
    • Add consultation orchestration guidance to vs-code-agents/planner.agent.md
  • Phase 3: Create Impact Analysis Templates

    • Create template for implementer impact analysis
    • Create template for architect impact analysis
    • Create template for security impact analysis
    • Create template for devops impact analysis
    • Create template for qa impact analysis
  • Phase 4: Update Specialist Agent Instructions

    • Update implementer agent with impact analysis capabilities (all 3 platforms)
    • Update architect agent with impact analysis capabilities (all 3 platforms)
    • Update security agent with impact analysis capabilities (all 3 platforms)
    • Update devops agent with impact analysis capabilities (all 3 platforms)
    • Update qa agent with impact analysis capabilities (all 3 platforms)
  • Phase 5: Documentation and Examples

    • Create example impact analysis workflow in CLAUDE.md
    • Update CLAUDE.md with impact analysis workflow
    • Update orchestrator with impact analysis routing patterns
    • Create comprehensive example (IMPACT-ANALYSIS-EXAMPLE.md)
    • Create framework overview (IMPACT-ANALYSIS-FRAMEWORK.md)
    • Update README.md with framework mention
  • Phase 6: Testing and Validation

    • Review all changes for consistency
    • Ensure synchronization across all three platforms
    • Final verification
    • Fix markdown code fence formatting issues

Recent Fix

Fixed malformed markdown code fence structures identified in code review:

  • Removed spurious ````text` fences that created nested fence issues
  • Added missing closing fences for markdown code blocks
  • Corrected template blocks to have proper opening and closing fences
  • Fixed "Phase" checklist sections to have proper fence structure

All 14 agent files updated with correct markdown formatting.

Original prompt

This section details on the original issue you should resolve

<issue_title>PRD: Multi-Agent Impact Analysis Framework for Planning Phase</issue_title>
<issue_description>## Product Requirements Document

Background

Currently, agent instructions in downstream projects (e.g., Qwiq) only document build configuration impact analysis. However, comprehensive change planning requires impact analysis across multiple dimensions: code structure, architecture, quality, security, and operations.

Reference: Qwiq PR rjmurillo/ai-agents#136 Discussion


Problem Statement

When planning changes, agents lack a structured framework for:

  1. Identifying which specialized agents should review proposed changes
  2. Conducting domain-specific impact analysis before implementation
  3. Ensuring all affected areas (code, architecture, quality, security, operations) are considered
  4. Documenting consultation results in a consistent format

This leads to:

  • Implementation without adequate cross-domain review
  • Missed downstream impacts (e.g., CI/CD breaks, security gaps, quality regressions)
  • Rework and delays when issues are discovered late
  • Inconsistent planning quality across projects

Proposed Solution

Add a Multi-Agent Impact Analysis Framework to the ai-agents system that enables the planner agent to orchestrate specialized agent consultations during the planning phase.

Core Capabilities

1. Agent Consultation Protocol

When planner agent creates a plan, it should:

  1. Analyze the proposed change scope
  2. Identify required specialist agent consultations
  3. Invoke each specialist agent with structured impact analysis prompts
  4. Aggregate consultation results
  5. Document findings in the plan

2. Specialist Agent Roles

Agent Type Impact Analysis Focus Key Questions
implementer Code structure, maintainability, patterns - Which files/modules need changes?
- Are there reusable patterns?
- What's the implementation complexity?
- Are there breaking changes?
architect Conceptual consistency, domain alignment - Does this align with domain model?
- Are abstractions consistent?
- Should an ADR be created?
- What are the long-term implications?
qa Quality, testability, edge cases - What's the test coverage strategy?
- What edge cases exist?
- How do we verify correctness?
- What regression risks exist?
security Security implications, vulnerabilities - Are there security implications?
- What's the threat model?
- Are there dependency vulnerabilities?
- What's the blast radius?
devops CI/CD, DX, operations - How does this affect CI/CD pipelines?
- What's the developer experience impact?
- Are there deployment considerations?
- Do validation scripts need updates?

3. Consultation Workflow

User Request → planner (scope analysis)
                   ↓
            [Identify required consultations]
                   ↓
         ┌─────────┴──────────┐
         ↓                    ↓
  implementer              architect
  (code impact)     (domain consistency)
         ↓                    ↓
    ┌────┴────┐          ┌───┴───┐
    ↓         ↓          ↓       ↓
   qa      security   devops  [other agents]
         ↓
  [Aggregate findings]
         ↓
  Final Plan Document

4. Impact Analysis Template

Each consultation should produce:

## {Agent} Impact Analysis

**Scope Reviewed**: [Brief description]

### Identified Impacts
- [ ] Impact 1: [Description] - Severity: High/Medium/Low
- [ ] Impact 2: [Description] - Severity: High/Medium/Low

### Affected Areas
- File/Component: [Details]
- File/Component: [Details]

### Recommendations
1. [Recommendation with rationale]
2. [Recommendation with rationale]

### Risks
- Risk 1: [Description + Mitigation]
- Risk 2: [Description + Mitigation]

### Follow-up Actions
- [ ] Action 1: [Assigned agent]
- [ ] Action 2: [Assigned agent]

User Stories

Story 1: Planner Orchestrates Multi-Agent Consultation

As a planner agent
I want to automatically identify and consult specialist agents
So that proposed changes are reviewed across all affected domains

Acceptance Criteria:

  • Given a change request scope, identify required specialist consultations
  • Invoke each specialist agent with structured prompts
  • Aggregate consultation results into final plan
  • Mark plan as "consultation-complete" when all specialists have reviewed

Story 2: Implementer Provides Code Impact Analysis

As an implementer agent
I want to analyze code-level impacts of proposed changes
So that architectural patterns, maintainability, and breaking changes are identified early

Acceptance Criteria:

  • Identify all files/modules requiring changes
  • Assess implementation comp...

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.


Note

Introduces a cross-platform Multi‑Agent Impact Analysis framework, updating planner/orchestrator/critic and specialist agents with workflows and templates, plus docs updates and removal of obsolete planning files.

  • Planning/Orchestration:
    • Framework: Add Multi-Agent Impact Analysis with triggers, consultation flow, aggregation, and metrics in planner across Claude, Copilot CLI, and VS Code.
    • Orchestrator: Add mandatory task classification/domain identification and impact-analysis orchestration and escalation paths.
    • Critic: Add impact-analysis validation and disagreement escalation to high-level-advisor.
  • Specialist Agents (all platforms):
    • Add Impact Analysis Mode + domain-specific checklists and output templates for implementer, architect, security, devops, qa (files in claude/, copilot-cli/, vs-code-agents/).
  • Docs:
    • Update CLAUDE.md, USING-AGENTS.md, README.md with framework overview, examples, and usage.
  • Cleanup:
    • Remove obsolete planning/critique docs under .agents/planning/ and .agents/critique/ (Phase 1/2 plans, PRD explainer, summary).

Written by Cursor Bugbot for commit da7efad. This will update automatically on new commits. Configure here.

Copilot AI and others added 2 commits December 15, 2025 20:22
Co-authored-by: rjmurillo <6811113+rjmurillo@users.noreply.github.com>
Co-authored-by: rjmurillo <6811113+rjmurillo@users.noreply.github.com>
Copilot AI changed the title [WIP] Add multi-agent impact analysis framework for planning phase Add Multi-Agent Impact Analysis Framework for planning phase Dec 15, 2025
Copilot AI requested a review from rjmurillo December 15, 2025 20:32
Addresses gaps identified in PRD review against GitHub issue #34:

**Planner Enhancements:**
- Add consultation status marker (In Progress | Complete | Blocked)
- Add inline metrics tracking (Coverage, Issues Discovered, Checkpoints)
- Add specialist disagreement handling with conflict documentation
- Note about critic escalation to high-level-advisor

**Specialist Template Additions:**
- Implementer: Breaking Changes section
- DevOps: Developer Experience Impact section
- Security: Blast Radius Assessment and Dependency Security sections
- Architect: Domain Model Alignment and Abstraction Consistency sections
- QA: Automation Strategy section
- All specialists: Issues Discovered tracking table

**Critic Agent Updates:**
- Impact Analysis Validation checklist
- Disagreement Detection & Escalation protocol
- Conflict Categories table for systematic assessment
- high-level-advisor added to Handoff Options

**Orchestrator Enhancements:**
- Impact Analysis Orchestration section with trigger conditions
- Orchestration flow for parallel specialist consultations
- Handling failed consultations and timeout strategy
- Escalation path table for disagreement resolution

**Infrastructure:**
- Fix 38 markdown fence errors across all agent files
- Create impact analysis framework review critique

Fixes #34

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements a Multi-Agent Impact Analysis Framework that enables comprehensive planning by orchestrating specialist agent consultations before implementation. The framework addresses the problem of missing cross-domain impact considerations during planning, which leads to late discovery of security, operations, quality, and architecture issues.

Key Changes:

  • Added impact analysis orchestration to planner agent with 5 trigger conditions and 3-phase consultation protocol
  • Enhanced 5 specialist agents (implementer, architect, security, devops, qa) with dedicated Impact Analysis Mode sections and domain-specific templates
  • Updated orchestrator with impact analysis routing patterns and escalation protocols
  • Updated critic with validation checklists for impact analysis completeness and specialist disagreement detection
  • Created comprehensive documentation (IMPACT-ANALYSIS-FRAMEWORK.md) and complete example walkthrough (IMPACT-ANALYSIS-EXAMPLE.md)

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
.agents/planning/IMPACT-ANALYSIS-FRAMEWORK.md New: Core framework documentation explaining process, benefits, and when to use
.agents/planning/IMPACT-ANALYSIS-EXAMPLE.md New: Complete OAuth 2.0 example demonstrating all 5 specialist consultations
.agents/critique/impact-analysis-framework-review.md New: Self-critique identifying partial implementations and gaps
claude/planner.md Added 247-line Multi-Agent Impact Analysis Framework section with templates
claude/orchestrator.md Added impact analysis orchestration flow with trigger conditions and escalation paths
claude/critic.md Added impact analysis validation checklist and disagreement detection protocol
claude/implementer.md Added Impact Analysis Mode with code impact template including breaking changes
claude/architect.md Added Impact Analysis Mode with architecture impact template including domain model alignment
claude/security.md Added Impact Analysis Mode with security impact template including blast radius
claude/devops.md Added Impact Analysis Mode with DevOps impact template including developer experience
claude/qa.md Added Impact Analysis Mode with QA impact template including automation strategy
vs-code-agents/*.agent.md Synchronized impact analysis additions with critical markdown fence bugs
copilot-cli/*.agent.md Synchronized impact analysis additions with critical markdown fence bugs
README.md Added reference to impact analysis example documentation
CLAUDE.md Added Impact Analysis Framework section with workflow examples

Critical Issue: The vs-code-agents/ and copilot-cli/ directories contain systematic markdown fence nesting errors throughout all modified files where ``` is immediately followed by ```text, creating malformed nested fences that will break markdown rendering. The claude/ directory uses the correct format and should be the reference for fixing these issues.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread vs-code-agents/planner.agent.md Outdated
Comment thread vs-code-agents/task-generator.agent.md Outdated
rjmurillo and others added 4 commits December 15, 2025 14:05
…chestrator

Enhances orchestrator with formal task classification and domain identification
to properly select agent sequences:

**Phase 0.5: Task Classification & Domain Identification (MANDATORY)**

Step 1: Classify Task Type
- 9 task types with definitions and signal words/patterns
- Feature, Bug Fix, Refactoring, Infrastructure, Security, Documentation,
  Research, Strategic, Ideation
- Classification output template with confidence and reasoning

Step 2: Identify Affected Domains
- 8 domains: Code, Architecture, Security, Operations, Quality, Data, API, UX
- Domain identification checklist
- Domain output template with primary/secondary domains and count

Step 3: Determine Complexity
- Maps task type + domain count to complexity level
- Single domain = Simple, 2 domains = Standard, 3+ = Complex
- Security/Strategic/Ideation always Complex

Step 4: Select Agent Sequence
- Classification Summary Template for documenting decisions

**Disagree and Commit Protocol**

Replaces timeout strategy with principled disagreement handling:
- Phase 1: Decision (dissent encouraged)
- Phase 2: Resolution (critic or high-level-advisor decides)
- Phase 3: Commitment (alignment required)
- Commitment language template
- Failure modes to avoid

Applied to all three orchestrator files (claude, copilot-cli, vs-code-agents).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add comprehensive documentation for the enhanced orchestrator workflow:

- Task Classification Framework: 9 task types with definitions and signal words
- Domain Identification Framework: 8 domains with scope descriptions
- Complexity Determination: mapping domain count to complexity level
- Impact Analysis Framework: when to use and consultation process
- Disagree and Commit Protocol: three-phase conflict resolution

Also:
- Fix duplicate heading names (MD024) for platform-specific sections
- Update workflow patterns with impact analysis variant
- Add Ideation Pipeline workflow
- Update agent descriptions with new capabilities

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@rjmurillo rjmurillo marked this pull request as ready for review December 15, 2025 22:19
Remove spurious fence pairs that created nested/malformed markdown
structures. The pattern of closing a fence with ``` then immediately
opening another ```text fence breaks markdown rendering.

Fixes:
- vs-code-agents/planner.agent.md: Multiple fence structure issues
  throughout the file (Memory Protocol, Planning Process phases,
  Impact Analysis Framework sections). Changed bold Phase headers
  to proper #### headings per MD036.
- vs-code-agents/task-generator.agent.md: Multiple fence structure
  issues in Memory Protocol, Decomposition Process, and template
  sections.

Addresses PR #40 review comments from @Copilot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@rjmurillo rjmurillo merged commit d9a0c20 into main Dec 15, 2025
6 checks passed
@rjmurillo rjmurillo deleted the copilot/add-impact-analysis-framework branch December 15, 2025 23:16
Copilot AI review requested due to automatic review settings March 23, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PRD: Multi-Agent Impact Analysis Framework for Planning Phase

3 participants