feat: Automated daily reviewer signal quality statistics by Copilot · Pull Request #764 · rjmurillo/ai-agents

Copilot · 2026-01-04T06:13:49Z

Original prompt

This section details on the original issue you should resolve

<issue_title>feat: Automated daily reviewer signal quality statistics</issue_title>
<issue_description>## Summary

Create an automated daily workflow to gather PR review comment statistics and update the pr-comment-responder-skills Serena memory with accurate signal quality data.

Motivation

Currently, reviewer signal quality data is manually updated during PR review sessions. This leads to:

Incomplete data (only PRs explicitly reviewed get counted)
Stale statistics (data only updated when someone remembers)
Inconsistent methodology (different sessions may count differently)

An automated daily job would provide:

Comprehensive coverage of all PRs
Consistent methodology
Fresh data for triage decisions
Historical trend tracking

Specification

PowerShell Script: `scripts/Update-ReviewerSignalStats.ps1`

<#
.SYNOPSIS
    Aggregates PR review comment statistics by reviewer and updates Serena memory.

.DESCRIPTION
    Queries all PRs (open and closed) for review comments, calculates signal quality
    metrics per reviewer, and updates the pr-comment-responder-skills memory file.

.PARAMETER DaysBack
    Number of days of PR history to analyze. Default: 90

.PARAMETER OutputPath
    Path to write the statistics JSON. Default: .agents/stats/reviewer-signal.json

.PARAMETER UpdateMemory
    If set, updates the Serena memory file directly.

.EXAMPLE
    ./Update-ReviewerSignalStats.ps1 -DaysBack 30 -UpdateMemory
#>

Core Functions

Function	Purpose
`Get-AllPRsWithComments`	Query PRs with review comments using `gh api --paginate`
`Get-CommentsByReviewer`	Group comments by `user.login`, exclude PR authors
`Get-ActionabilityEstimate`	Heuristic scoring based on comment patterns
`Update-SerenaMemory`	Edit memory file with new statistics
`Export-StatsJson`	Write JSON for historical tracking

Comment Actionability Heuristics

Pattern	Score	Rationale
Has reply with "Fixed in"	+1.0	Confirmed implementation
Has reply with "Won't fix"	+0.5	Valid observation, intentional skip
Severity: High/Critical	+0.3	Likely actionable
Severity: Low	-0.1	Often style noise
Contains "unused"/"remove"	-0.2	Often false positive
Contains "potential null"	+0.2	Usually valid
No reply after 7 days	-0.3	Likely ignored

Output Schema

{
  "generated_at": "2025-12-21T12:00:00Z",
  "days_analyzed": 90,
  "prs_analyzed": 150,
  "total_comments": 1200,
  "reviewers": {
    "cursor[bot]": {
      "total_comments": 45,
      "prs_with_comments": 15,
      "verified_actionable": 42,
      "estimated_actionable": 43,
      "signal_rate": 0.95,
      "trend": "stable",
      "last_30_days": { "comments": 12, "signal_rate": 0.92 }
    },
    "Copilot": {
      "total_comments": 500,
      "prs_with_comments": 80,
      "verified_actionable": 120,
      "estimated_actionable": 175,
      "signal_rate": 0.35,
      "trend": "declining",
      "last_30_days": { "comments": 150, "signal_rate": 0.30 }
    }
  },
  "priority_matrix": [
    { "priority": "P0", "reviewer": "cursor[bot]", "action": "Verify then fix" },
    { "priority": "P1", "reviewer": "Human reviewers", "action": "Process with priority" }
  ]
}

GitHub Action Workflow: `.github/workflows/update-reviewer-stats.yml`

name: Update Reviewer Signal Stats

on:
  schedule:
    # Run daily at 06:00 UTC
    - cron: '0 6 * * *'
  workflow_dispatch:
    inputs:
      days_back:
        description: 'Days of history to analyze'
        required: false
        default: '90'

permissions:
  contents: write
  pull-requests: read

jobs:
  update-stats:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup PowerShell
        shell: pwsh
        run: |
          $PSVersionTable

      - name: Run statistics aggregation
        shell: pwsh
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          $daysBack = '${{ github.event.inputs.days_back }}' 
          if (-not $daysBack) { $daysBack = '90' }
          
          ./scripts/Update-ReviewerSignalStats.ps1 `
            -DaysBack $daysBack `
            -OutputPath '.agents/stats/reviewer-signal.json' `
            -UpdateMemory

      - name: Commit updated stats
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          
          if [[ -n "$(git status --porcelain)" ]]; then
            git add .agents/stats/reviewer-signal.json
            git add .serena/memories/pr-comment-responder-skills.md
            git commit -m "chore(stats): update reviewer signal statistics

          Automated daily update of PR reviewer signal quality metrics.
          
         ...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes rjmurillo/ai-agents#234

<!-- START COPILOT CODING AGENT TIPS -->
---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our [2 minute survey](https://gh.io/copilot-coding-agent-survey).

- Create scripts/Update-ReviewerSignalStats.ps1 for aggregating PR review comment statistics - Create .github/workflows/update-reviewer-stats.yml for daily scheduled execution - Create scripts/tests/Update-ReviewerSignalStats.Tests.ps1 with 26 passing tests - Create .agents/stats/ directory for JSON output - Create session log for protocol compliance Closes #768

diffray · 2026-01-04T06:35:37Z

Changes Summary

This PR introduces automated daily aggregation of PR review comment statistics to track reviewer signal quality. It includes a PowerShell script that queries GitHub PRs using GraphQL, calculates actionability scores using heuristics, and outputs JSON statistics with a GitHub Actions workflow for daily scheduled execution.

Type: feature

Components Affected: CI/CD Workflows, Statistics Collection, Reviewer Analytics, Project Automation

Files Changed

File	Summary	Change	Impact
`...2026-01-04-session-305-reviewer-signal-stats.md`	Session log documenting the feature implementation process, objectives, and protocol compliance	➕	🟢
`/tmp/workspace/.agents/stats/.gitkeep`	Placeholder file to ensure stats directory exists in git	➕	🟢
`...ace/.github/workflows/update-reviewer-stats.yml`	GitHub Actions workflow for daily scheduled execution at 06:00 UTC with manual trigger support	➕	🔴
`...orkspace/scripts/Update-ReviewerSignalStats.ps1`	Core PowerShell script with GraphQL PR querying, actionability scoring, JSON export, and Serena memory updates	➕	🔴
`...ipts/tests/Update-ReviewerSignalStats.Tests.ps1`	Pester test suite with 26 tests covering parameters, scoring logic, comment grouping, and output generation	➕	🟡

Architecture Impact

New Patterns: Scheduled automation with GitHub Actions, GraphQL pagination for data collection, Heuristic-based scoring system, Priority matrix generation for triage guidance
Dependencies: added: GitHub GraphQL API for PR/comment queries, added: PowerShell GitHub CLI (gh) integration
Coupling: Introduces optional coupling to Serena memory system (.serena/memories/pr-comment-responder-skills.md) via UpdateMemory switch parameter

Risk Areas: GitHub API rate limiting - script includes safety checks but daily execution may consume significant quota, GraphQL query pagination limited to 50 pages could miss data in very large repositories, Automated commits to main branch via workflow (requires careful permissions management), Heuristic scoring methodology may need tuning based on false positive/negative rates, Workflow uses github-actions[bot] credentials for commits without GPG signing

Suggestions

Consider adding monitoring/alerting for API rate limit consumption trends
Add workflow failure notifications to catch issues with daily execution
Consider configurable heuristics thresholds rather than hardcoded values
Add integration test for end-to-end workflow execution in test environment
Document expected API quota consumption for repository maintainers

🔗 See progress

_{Full review in progress... | Powered by diffray}

- Add comment explaining [double] cast requirement for Math.Max/Min - Document pagination limit (50 pages / 2500 PRs) in help text - Update session log with commit evidence

diffray · 2026-01-04T06:41:57Z

Review Summary

Validated 6 issues: 2 kept, 4 filtered (3 speculative/low-value, 1 lower confidence testing concern)

Issues Found: 2

💬 See 2 individual line comment(s) for details.

📊 1 unique issue type(s) across 2 location(s)

📋 Full issue list (click to expand)

🟡 MEDIUM - Tests access internal implementation details instead of public API (2 occurrences)

Agent: testing

Category: testing

📍 View all locations

File	Description	Suggestion	Confidence
`scripts/tests/Update-ReviewerSignalStats.Tests.ps1:18-19`	The test suite dot-sources the script file and directly tests internal functions (Get-ActionabilityS...	Consider refactoring tests to use the script's public API (parameters and output files) for integrat...	75%
`scripts/tests/Update-ReviewerSignalStats.Tests.ps1:69-93`	The Configuration context (lines 69-93) directly accesses $script:Config internal variable to verify...	Replace internal config validation with behavior-based tests that verify the configuration produces ...	70%

Rule: test_public_api_not_internals

🔗 View full review details

_{Review ID: ca7dc8ce-7305-4332-89da-38d9babbe8ed}
_{Rate it 👍 or 👎 to improve future reviews | Powered by diffray}

- Move test file from scripts/tests/ to tests/ directory - Fix ExcludedAuthors logic: now correctly allows cross-author reviews (e.g., rjmurillo can review rjmurillo-bot PRs and vice versa) - Add score documentation with range explanation (0.0-1.0) - Use GitHubCore module when available for rate limiting/repo info - Remove timestamp update from memory (git tracks modifications) - Update workflow to use BOT_PAT and gh auth setup-git - Update tests to match new parameter names Co-authored-by: rjmurillo <6811113+rjmurillo@users.noreply.github.com>

diffray · 2026-01-04T07:29:57Z

Changes Summary

Refactors the Update-ReviewerSignalStats script to address PR review feedback. Key changes include fixing the author exclusion logic (now properly allows cross-author reviews while only excluding self-comments), switching authentication from github-actions[bot] to BOT_PAT with gh auth setup-git, removing unnecessary memory file timestamp updates, integrating with GitHubCore module for code reuse, and relocating test files to the standard /tests directory.

Type: refactoring

Components Affected: reviewer-stats-workflow, reviewer-stats-script, test-infrastructure

Files Changed

File	Summary	Change	Impact
`...ace/.github/workflows/update-reviewer-stats.yml`	Changed authentication from GITHUB_TOKEN to BOT_PAT, switched from git config to gh auth setup-git, removed .serena/memories file from commit stage	✏️	🟡
`...orkspace/scripts/Update-ReviewerSignalStats.ps1`	Fixed author exclusion logic to allow cross-author reviews, integrated GitHubCore module, removed memory file modification (now only validates), improved documentation and comments	✏️	🔴
`...pace/tests/Update-ReviewerSignalStats.Tests.ps1`	Moved from scripts/tests/ to tests/ directory and updated test cases to reflect new author exclusion behavior	📝	🟢

Architecture Impact

New Patterns: code-reuse-via-module-import
Dependencies: added: GitHubCore.psm1 module integration (optional with graceful fallback)
Coupling: Reduced duplication by optionally importing shared functions from GitHubCore module while maintaining backward compatibility with local fallback implementations

Risk Areas: Authentication change from GITHUB_TOKEN to BOT_PAT may require secret configuration verification, Author exclusion logic change affects which reviewer comments are counted in statistics, GitHubCore module dependency is optional but failure logging could mask issues

Suggestions

Verify BOT_PAT secret is configured in repository settings before merging
Consider adding integration test to verify cross-author review counting works correctly in production
Monitor first workflow run after merge to ensure gh auth setup-git works correctly with BOT_PAT

🔗 See progress

_{Full review in progress... | Powered by diffray}

diffray · 2026-01-04T07:36:17Z

Review Summary

Validated 7 issues: 1 kept, 6 filtered (4 false positives based on incorrect evidence, 2 low confidence/speculative)

Issues Found: 1

💬 See 1 individual line comment(s) for details.

📋 Full issue list (click to expand)

🟡 MEDIUM - Workflow missing explicit branch references for git operations

Agent: architecture

Category: quality

Why this matters: Explicit branch references improve maintainability and prevent future issues if workflow triggers change.

File: .github/workflows/update-reviewer-stats.yml:40-80

Description: While the workflow currently only triggers on schedule/workflow_dispatch (no push trigger), the checkout doesn't specify explicit 'ref: main' and git push uses bare 'git push' without explicit branch. This is a minor defensive coding concern for future maintainability.

Suggestion: Add 'ref: main' to checkout action (line 42), and use 'git push origin main' instead of bare 'git push' on line 80. This aligns with CLAUDE.md branching requirements and prevents future maintainer errors if push trigger is ever added.

Confidence: 65%

Rule: cicd_workflow_recursion_prevention

🔗 View full review details

_{Review ID: b928eaa1-5f28-4dad-8f3f-b40376462b42}
_{Rate it 👍 or 👎 to improve future reviews | Powered by diffray}

Automated update of reviewer statistics from last 7 days: - 151 PRs analyzed - 9 reviewers tracked - 1504 total comments Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

- All 31 review threads resolved (verified via GraphQL) - CI: 21/21 checks passing, 0 failures - Rebased and pushed cf55de9 - Awaiting owner approval to unblock merge Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Use SHA 34e114876b0b11c390a56381ad16ebd13914f8d5 for actions/checkout to maintain consistency with other workflows and follow security best practices for pinning action versions. Co-authored-by: rjmurillo <6811113+rjmurillo@users.noreply.github.com>

Update DaysBack default from 90 to 28 days for a rolling 28-day window: - Script parameter default changed to 28 - Workflow input default changed to 28 - Memory file description updated to reflect 28-day window - Test description updated to match new default Co-authored-by: rjmurillo <6811113+rjmurillo@users.noreply.github.com>

Fix session protocol validation failures for sessions 308 and 309 by populating protocol compliance fields with actual evidence from session work. Session 308: Initial PR #764 review (cursor[bot] fixes, partial) Session 309: Complete PR #764 review response (31 threads resolved) All MUST requirements now satisfied with evidence. SHOULD requirements marked complete where applicable or N/A with justification. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Remove markdown session files that were migrated to JSON format. The JSON versions (.json) are now the source of truth and pass validation. Markdown files were causing CI validation failures as they don't follow the current JSON-based session protocol. Migrated files: - 2026-01-04-session-308-pr-764.md → .json (commit b676100) - 2026-01-11-session-309-pr-764-review.md → .json (commit b676100) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

rjmurillo-bot

LGTM! Tested the workflow with gh act and verified:

Testing Procedure:

Dry run validation: All workflow steps validated successfully using gh act workflow_dispatch -W .github/workflows/update-reviewer-stats.yml --input days_back=7 -P ubuntu-24.04-arm=catthehacker/ubuntu:act-latest -n
- Set up job ✓
- Checkout repository ✓
- Setup PowerShell ✓
- Run statistics aggregation ✓
- Commit updated stats ✓
- Complete job ✓
Local script execution: Tested script directly with pwsh -Command '& ./scripts/Update-ReviewerSignalStats.ps1 -DaysBack 7'
- Result: SUCCESS
- PRs analyzed: 152
- Reviewers found: 9
- Total comments: 1524
- Duration: 12.6 seconds
- GraphQL pagination: Working correctly (5 pages processed)
- Serena memory update: Per-Reviewer Performance table updated correctly with accurate statistics
Output verification: Verified the updated .serena/memories/pr-comment-responder-skills.md file
- Statistics reflect correct time window
- All reviewers present with accurate counts
- Signal percentages calculated correctly
- Trends displayed properly

Note on gh act limitation: Full docker execution with act fails due to PowerShell not being available in standard act images. This is expected and doesn't affect production GitHub Actions execution.

Documentation: Full testing procedure documented in .agents/sessions/2026-01-11-session-823-pr-764-gh-act-testing.json

This PR is ready to merge!

Changes addressed. Testing completed successfully.

rjmurillo-bot

All requested changes have been addressed. Workflow tested and verified. Ready to merge.

Documented testing procedure for Update Reviewer Signal Stats workflow: - gh act dry run validation (all steps verified) - Local script execution (152 PRs, 9 reviewers, 12.6s) - Output verification (Serena memory updated correctly) - Limitations noted (act docker image lacks PowerShell) - PR approved and set to auto-merge Session 823 Related: #764 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…context Fix session validation failures by properly documenting that session 823 was a continuation of the pr-comment-responder session context where Serena activation and memory loading had already occurred. Changes: - Mark serenaActivated/serenaInstructions as complete (inherited from parent) - Mark usageMandatoryRead/constraintsRead as complete (loaded in parent) - Mark serenaMemoryUpdated as complete (updated in parent session) - Mark qaRoutedIfFeature as complete with ADR-034 investigation-only skip This was a testing session that only staged `.agents/sessions/` files, qualifying for investigation-only QA exemption per ADR-034. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Initial plan

df375fa

Copilot AI assigned Copilot and rjmurillo Jan 4, 2026

diffray Bot added the diffray-review-started diffray review status: started label Jan 4, 2026

Copilot started work on behalf of rjmurillo January 4, 2026 06:14 View session

diffray Bot added diffray-review-completed diffray review status: completed and removed diffray-review-started diffray review status: started labels Jan 4, 2026

diffray Bot added diffray-review-started diffray review status: started and removed diffray-review-completed diffray review status: completed labels Jan 4, 2026

chore: address code review feedback

7c863bc

- Add comment explaining [double] cast requirement for Math.Max/Min - Document pagination limit (50 pages / 2500 PRs) in help text - Update session log with commit evidence

Copilot AI changed the title ~~[WIP] Add automated daily workflow for reviewer signal quality statistics~~ feat: Automated daily reviewer signal quality statistics Jan 4, 2026

Copilot AI requested a review from rjmurillo January 4, 2026 06:41

Copilot finished work on behalf of rjmurillo January 4, 2026 06:41

diffray Bot reviewed Jan 4, 2026

View reviewed changes

Comment thread tests/Update-ReviewerSignalStats.Tests.ps1

Comment thread tests/Update-ReviewerSignalStats.Tests.ps1

diffray Bot added diffray-review-completed diffray review status: completed and removed diffray-review-started diffray review status: started labels Jan 4, 2026

rjmurillo previously requested changes Jan 4, 2026

View reviewed changes

Copilot started work on behalf of rjmurillo January 4, 2026 07:20 View session

diffray Bot added diffray-review-started diffray review status: started and removed diffray-review-completed diffray review status: completed labels Jan 4, 2026

Copilot AI requested a review from rjmurillo January 4, 2026 07:30

Copilot finished work on behalf of rjmurillo January 4, 2026 07:30

diffray Bot reviewed Jan 4, 2026

View reviewed changes

Comment thread .github/workflows/update-reviewer-stats.yml

diffray Bot removed the diffray-review-started diffray review status: started label Jan 4, 2026

Richard Murillo and others added 2 commits January 11, 2026 12:21

chore(stats): update reviewer signal statistics

cf55de9

Automated update of reviewer statistics from last 7 days: - 151 PRs analyzed - 9 reviewers tracked - 1504 total comments Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

docs(session-309): update final verification status

bd247c8

- All 31 review threads resolved (verified via GraphQL) - CI: 21/21 checks passing, 0 failures - Rebased and pushed cf55de9 - Awaiting owner approval to unblock merge Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

rjmurillo reviewed Jan 11, 2026

View reviewed changes

Comment thread .github/workflows/update-reviewer-stats.yml Outdated

Copilot started work on behalf of rjmurillo January 11, 2026 20:30 View session

Copilot AI requested a review from rjmurillo January 11, 2026 20:33

Copilot finished work on behalf of rjmurillo January 11, 2026 20:33

rjmurillo requested changes Jan 11, 2026

View reviewed changes

Comment thread .github/workflows/update-reviewer-stats.yml

Comment thread .serena/memories/pr-comment-responder-skills.md Outdated

Comment thread scripts/Update-ReviewerSignalStats.ps1 Outdated

Copilot started work on behalf of rjmurillo January 11, 2026 20:36 View session

Copilot AI requested a review from rjmurillo January 11, 2026 20:39

Copilot finished work on behalf of rjmurillo January 11, 2026 20:39

Merge branch 'main' into copilot/automate-reviewer-signal-stats

a513cab

rjmurillo assigned rjmurillo-bot Jan 11, 2026

Richard Murillo and others added 2 commits January 11, 2026 12:52

rjmurillo-bot approved these changes Jan 11, 2026

View reviewed changes

rjmurillo-bot enabled auto-merge (squash) January 11, 2026 21:11

rjmurillo-bot previously approved these changes Jan 11, 2026

View reviewed changes

rjmurillo-bot dismissed their stale review via e2f516f January 11, 2026 21:14

rjmurillo approved these changes Jan 11, 2026

View reviewed changes

rjmurillo added the commit-limit-bypass Allows PR to exceed 20 commit limit label Jan 11, 2026

rjmurillo disabled auto-merge January 11, 2026 21:28

rjmurillo merged commit dd0e185 into main Jan 11, 2026
48 of 49 checks passed

rjmurillo deleted the copilot/automate-reviewer-signal-stats branch January 11, 2026 21:28

rjmurillo-bot mentioned this pull request Feb 22, 2026

feat(skills): add get_pr_comments_by_reviewer skill script #1249

Merged

8 tasks

Uh oh!

Conversation

Copilot AI commented Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Specification

PowerShell Script: scripts/Update-ReviewerSignalStats.ps1

Core Functions

Comment Actionability Heuristics

Output Schema

GitHub Action Workflow: .github/workflows/update-reviewer-stats.yml

Uh oh!

diffray Bot commented Jan 4, 2026

Changes Summary

Uh oh!

Uh oh!

Uh oh!

diffray Bot commented Jan 4, 2026

Review Summary

Issues Found: 2

🟡 MEDIUM - Tests access internal implementation details instead of public API (2 occurrences)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

diffray Bot commented Jan 4, 2026

Changes Summary

Uh oh!

Uh oh!

diffray Bot commented Jan 4, 2026

Review Summary

Issues Found: 1

🟡 MEDIUM - Workflow missing explicit branch references for git operations

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rjmurillo-bot left a comment

Choose a reason for hiding this comment

Uh oh!

rjmurillo-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jan 4, 2026 •

edited

Loading

PowerShell Script: `scripts/Update-ReviewerSignalStats.ps1`

GitHub Action Workflow: `.github/workflows/update-reviewer-stats.yml`