Skip to content

Plugin Directory: Add automated plugin review job#586

Open
obenland wants to merge 9 commits intoWordPress:trunkfrom
obenland:add/plugin-directory-automated-review
Open

Plugin Directory: Add automated plugin review job#586
obenland wants to merge 9 commits intoWordPress:trunkfrom
obenland:add/plugin-directory-automated-review

Conversation

@obenland
Copy link
Copy Markdown
Member

@obenland obenland commented Apr 2, 2026

Summary

  • Introduces a 3-phase automated review system (triage → batch → synthesis) that reviews plugin submissions against directory guidelines, security checklists, and code quality standards using the WordPress AI Client API
  • Bundled prompt references with review rules encoding experienced reviewer judgment, plus fallback to bundled guidelines when DevHub is unreachable
  • Admin UI button gated to review-eligible statuses with proper nonce/cap checks; integrates with existing Plugin Check (PCP) results for cross-referencing
  • Plugin source files wrapped in XML tags with randomized boundary tokens to mitigate prompt injection
  • 39 unit tests covering verdict logic, fallback aggregation, batch building, file collection, triage heuristics, PCP integration, and output formatting

Test plan

  • Verify automated review triggers on plugin upload (cron job queued after 1 hour)
  • Verify "Run Automated Review" button appears only on draft/pending/new plugins when wp_supports_ai() is available
  • Verify AJAX handler rejects missing slug, invalid nonce, and unauthorized users
  • Verify review results appear in Internal Notes with correct verdict/findings format
  • Verify fallback to bundled guidelines when DevHub is unreachable
  • Verify verdict enforcement: blockers always produce "reject" regardless of AI response
  • Verify incomplete batch coverage (skipped/failed batches) prevents "approve"
  • Run unit tests: cd environments && npm run plugins:test

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings April 2, 2026 14:59
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 2, 2026

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props obenland.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an automated (AI-assisted) review job to the Plugin Directory submission workflow, aiming to produce a structured verdict and findings for reviewer internal notes using a triage → batch → synthesis loop.

Changes:

  • Introduces a new cron/AJAX-driven job (Plugin_Automated_Review) that extracts the uploaded ZIP, batches source files, calls the WordPress AI Client API, and logs/stores results.
  • Wires the job into existing job infrastructure (colon-based cron hooks) and plugin upload flow, plus adds an admin metabox button to trigger reviews on-demand.
  • Adds bundled prompt/reference markdown files, including a guidelines fallback when DevHub content can’t be fetched.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
jobs/class-plugin-automated-review.php Implements the multi-phase automated review loop, batching, AI calls, result storage, and audit logging.
jobs/class-manager.php Registers the new colon-based cron task handler mapping and queues review on plugin_upload.
jobs/automated-review/triage-prompt.md Triage system prompt template.
jobs/automated-review/batch-prompt.md Batch review system prompt template + security/guideline rules.
jobs/automated-review/synthesis-prompt.md Synthesis system prompt template for final report.
jobs/automated-review/guidelines.md Bundled guidelines fallback when DevHub is unreachable.
admin/metabox/class-review-tools.php Adds “Run Automated Review” button + AJAX call in the review metabox.
admin/class-customizations.php Registers the new admin-ajax action for on-demand review runs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Introduces a 3-phase automated review system (triage → batch → synthesis) that
reviews plugin submissions against directory guidelines, security checklists,
and code quality standards using the WordPress AI Client API.

- New `Plugin_Automated_Review` job class with cron and AJAX entry points
- Bundled prompt references: security checklist, guidelines, 13 micro-prompts
  encoding experienced reviewer judgment rules
- Integrates with existing Plugin Check (PCP) results for cross-referencing
- Falls back to bundled guidelines when DevHub is unreachable
- Enforces verdict consistency (blockers always mean reject)
- Admin UI button gated to review-eligible statuses with proper nonce/cap checks
- WPCS formatting cleanup in class-review-tools.php (short → long array syntax)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@obenland obenland force-pushed the add/plugin-directory-automated-review branch from 0c2e01a to c87848e Compare April 2, 2026 15:09
- Pass slug args to wp_next_scheduled() so duplicate detection works
- Check wp_supports_ai() in cron_trigger() and async AJAX path
- Cast file_get_contents() returns to string to avoid TypeError with strict_types
- Wrap AI client call in try/finally to ensure timeout filter is always removed

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

obenland and others added 2 commits April 2, 2026 11:26
- Escape file paths in XML tag attributes with esc_attr()
- Add untrusted-content warning to triage system prompt
- Cast batch-prompt.md file_get_contents to string for strict_types safety

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix collect_files() to skip vendor/ and node_modules/ inside nested
  plugin directories (e.g., my-plugin/vendor/) not just at the root
- Extract determine_verdict() as single source of truth for verdict logic,
  used by both the AI synthesis path and the fallback path
- Remove unreachable verdict check (build_fallback_result always sets it)
- Incomplete batch coverage now adds a warning finding so determine_verdict()
  naturally produces needs_changes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@obenland obenland force-pushed the add/plugin-directory-automated-review branch from f76caa7 to a32e8c5 Compare April 2, 2026 15:37
@obenland obenland requested a review from Copilot April 2, 2026 15:37
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@obenland obenland force-pushed the add/plugin-directory-automated-review branch from 22f0274 to bb7004e Compare April 2, 2026 16:49
- Use end() for ZIP attachment selection instead of max(array_keys())
  for more reliable "newest attachment" semantics
- Include vendor/node_modules paths in all_files list (still excluded
  from source_files) so structure analysis can detect dev artifacts
- Pass full file list to batch prompts for disallowed-file detection
- Rewrite synthesis prompt to request JSON output matching the schema,
  fixing mismatch with as_json_response() that could force fallback path
- Restore leading backslash on use import to match file convention

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@obenland obenland force-pushed the add/plugin-directory-automated-review branch from bb7004e to ec81148 Compare April 2, 2026 16:50
@obenland obenland requested a review from Copilot April 2, 2026 16:51
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…tches

- Inject incomplete-coverage warning in run_review() before
  determine_verdict(), covering both AI synthesis and fallback paths
- Remove duplicate incomplete-coverage logic from build_fallback_result()
- Cast guidelines fallback file_get_contents to (string)
- Fix batch prompt: replace per-guideline PASS/FAIL/WARN/N/A instruction
  with "only create findings for violations" to match JSON schema
- Mark common-issues section as internal reference only to prevent
  fix suggestions from leaking into findings output

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

39 tests covering the deterministic logic methods:
- determine_verdict(): blockers→reject, warnings→needs_changes, clean→approve
- build_fallback_result(): severity bucketing, field defaults, multi-batch
  aggregation, unknown severity routing
- normalize_file_priorities(): array-of-objects and map form handling
- build_default_triage(): extension-based priority, PCP error promotion
- build_batches(): skip filtering, oversized isolation, priority sorting,
  batch size splitting
- collect_files(): vendor/node_modules skipping, extension filtering
- summarize_pcp_results(): error/warning counting, formatted output
- format_as_note(): verdict display, HTML output, finding rendering
- format_pcp_for_file(): forward and reverse path matching
- find_readme_content(): regex matching

Pure-logic methods made public for direct testability. Uses Yoast
PHPUnit Polyfills XTestCase for PHPUnit 10 compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@obenland obenland force-pushed the add/plugin-directory-automated-review branch from 411facf to d245ca6 Compare April 3, 2026 15:51
Catch missing or invalid attachment file paths early with an explicit
check, rather than relying on Filesystem::unzip() to fail gracefully.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@obenland obenland force-pushed the add/plugin-directory-automated-review branch from d245ca6 to 965c707 Compare April 3, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants