chore: add /codebase-audit skill for deep parallel codebase auditing#613
chore: add /codebase-audit skill for deep parallel codebase auditing#613
Conversation
Adds a reusable Claude Code skill that orchestrates 20+ specialized agents to audit the entire codebase (or targeted scope) for issues. Key features: - Architecture research phase BEFORE launching audit agents - Parallel agent swarm (each agent searches for one issue type) - Mandatory validation pass (32% false positive rate observed) - Skepticism for "100% clean" results triggers deeper investigation - Work package grouping by code proximity, not severity - Existing issue deduplication (twice: in prompts + before creation) - Configurable scope (full, src/, web/, cli/, docs/, .github/) Methodology refined from a battle-tested audit session that produced 30 validated findings across 11 work packages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a powerful new Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughAdds a new Claude skill at 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. 📝 Coding Plan
Comment |
There was a problem hiding this comment.
Code Review
This pull request introduces a very thorough and well-structured /codebase-audit skill. The multi-phase approach, emphasizing context-gathering, mandatory validation, and skepticism, is excellent. The developer-centric grouping of findings by code proximity is also a great feature. My feedback focuses on improving the clarity, consistency, and logical flow of the skill's definition, particularly around its arguments and conditional steps.
| @@ -0,0 +1,395 @@ | |||
| --- | |||
| description: "Deep codebase audit: launches specialized parallel agents to find issues, validates findings, groups into work packages, and creates GitHub issues" | |||
| argument-hint: "<scope: full | src/ | web/ | cli/ | docs/ | .github/> [--report-only] [--skip-issues]" | |||
There was a problem hiding this comment.
To improve clarity and avoid redundancy, it's best to have a single flag for a given behavior. Both --report-only and --skip-issues (defined on line 36) have the same effect. I recommend removing --skip-issues and using --report-only as the canonical flag.
| argument-hint: "<scope: full | src/ | web/ | cli/ | docs/ | .github/> [--report-only] [--skip-issues]" | |
| argument-hint: "<scope: full | src/ | web/ | cli/ | docs/ | .github/> [--report-only]" |
| | `docs/` or `site/` | Documentation/site | Docs/content categories | | ||
| | `.github/` or `ci` | CI/CD only | CI/workflow categories | | ||
| | `--report-only` | Any scope | Skip issue creation, report only | | ||
| | `--skip-issues` | Any scope | Same as --report-only | |
There was a problem hiding this comment.
This argument is redundant with --report-only. To simplify the skill's interface and align with the change suggested for line 3, it's best to remove this line.
| | `test-coverage` | Public modules with no corresponding test file, empty test files | | ||
| | `flaky-tests` | Unmocked time, real asyncio.sleep in tests, timing-dependent assertions, skipped tests | | ||
| | `wiring-lifecycle` | Incorrectly wired services, missing DI, lifecycle gaps, protocol implementations incomplete | | ||
| | `security-gaps` | Hardcoded secrets, missing auth guards, injection vectors, SSRF, XSS | |
There was a problem hiding this comment.
The security-gaps agent is listed under "Python Backend Agents", but its search scope ("Hardcoded secrets, missing auth guards, injection vectors, SSRF, XSS") includes issues that are cross-cutting and can appear in frontend, infrastructure, or CLI code.
Consider either:
- Moving
security-gapsto the "Cross-Cutting Agents" section to reflect its broad scope. - Creating separate, stack-specific security agents (e.g., for Vue, Go, Docker) to ensure comprehensive security coverage.
| 4. **Validate any new findings** (same as Phase 4) | ||
| 5. Add validated findings to the main list | ||
|
|
||
| Skip this phase if the user passed `--quick` or if the zero-finding categories are genuinely well-covered (e.g., dependencies audit finding nothing is believable). |
There was a problem hiding this comment.
This line mentions a --quick flag, but it is not defined in the skill's arguments in Phase 0 (line 3 and the table on lines 27-36). For the skill to be complete and consistent, please add --quick to the argument-hint and the arguments table, describing its purpose of skipping this deep-dive phase.
| Ask the user: | ||
| 1. **"Proceed to group into work packages and create issues" (Recommended)** | ||
| 2. "Show me the full detail for each finding first" | ||
| 3. "Export as markdown report only (no issues)" |
There was a problem hiding this comment.
The user gate offers an option to "create issues", but this doesn't account for the user having already passed the --report-only flag at the start. The skill's logic should be conditional. If --report-only was specified, this user gate should be skipped or should not present options related to creating issues.
| |---|---------| | ||
| | 1 | ... | | ||
| | 2 | ... | | ||
| **Rationale:** Why these go together. |
There was a problem hiding this comment.
This **Rationale:** line is inside the example code block, but it breaks the markdown table format and seems misplaced. It should likely be regular text outside of the code block, following the table for each work package, to ensure the example is valid.
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.claude/skills/codebase-audit/SKILL.md:
- Line 3: The argument hint string (argument-hint) is incomplete and
inconsistent with later phases; update the argument-hint value so it lists all
supported scopes and flags referenced elsewhere (e.g., include ci, docker/,
site/, --quick, --report-only, --skip-issues) so the CLI/help is discoverable
and consistent across phases; locate the argument-hint entries (the literal
"argument-hint" lines at 3, 33-34, 118, 256-257 in this file) and standardize
them to a single canonical set of options used throughout the skill
documentation.
- Line 4: The allowed-tools list currently includes write-capable tools ("Edit",
"Write") which contradicts the audit-only constraint described later; modify the
SKILL.md allowed-tools declaration to remove any mutable/write-capable entries
(specifically remove "Edit" and "Write") so the skill only permits read-only
tools (e.g., "Read", "Grep", "Glob", "WebFetch", etc.), and ensure the updated
list aligns with the later prohibition on fixing code referenced around the rule
at line 395.
- Line 13: Update the wording at the top rule that currently reads “validation
is mandatory before ANY output to the user” to the proposed clearer phrase
“Never present unvalidated findings” so it no longer conflicts with the later
guidance in the document (specifically the guidance around pre-validation
reporting at lines referencing “pre-validation reporting”); replace the exact
sentence and ensure any nearby references that repeat the stricter phrasing are
updated to match the new wording to avoid contradiction.
- Line 143: The markdown file contains unlabeled fenced code blocks causing
markdownlint MD040 failures; update each unlabeled triple-backtick fence in
.claude/skills/codebase-audit/SKILL.md to include an explicit language
identifier (e.g., ```md or ```text) for every occurrence flagged (the fences
around the sample table and the other four unlabeled blocks), keeping the
existing fence contents unchanged and only adding the language token after the
opening backticks so linting and syntax highlighting work correctly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 54d54983-9314-4172-8aba-7afd57629a6b
📒 Files selected for processing (1)
.claude/skills/codebase-audit/SKILL.md
…sistency agent - Remove Edit/Write from allowed-tools (contradicts audit-only rule 12) - Consolidate argument-hint: add missing scopes (ci, docker/, site/), add --quick flag, remove redundant --skip-issues - Remove --skip-issues from Phase 0 table (redundant with --report-only) - Add --quick flag definition to Phase 0 table - Soften validation wording to avoid contradiction with progress reporting - Move security-gaps agent from Python Backend to Cross-Cutting Agents - Make Phase 6 user gate conditional on --report-only flag - Fix Rationale line placement (was inside code block, now outside) - Add language identifiers to all unlabeled fenced code blocks (MD040) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
.claude/skills/codebase-audit/SKILL.md (1)
3-3:⚠️ Potential issue | 🟡 MinorAlign
argument-hintwith documentedsrc/synthorg/scope.
Phase 0documentssrc/synthorg/as valid, butargument-hintomits it, making help text inconsistent.Suggested doc alignment
-argument-hint: "<scope: full | src/ | web/ | cli/ | docs/ | site/ | .github/ | ci | docker/> [--report-only] [--quick]" +argument-hint: "<scope: full | src/ | src/synthorg/ | web/ | cli/ | docs/ | site/ | .github/ | ci | docker/> [--report-only] [--quick]"Also applies to: 30-30
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.claude/skills/codebase-audit/SKILL.md at line 3, Update the argument-hint string to include the documented src/synthorg/ scope so help text matches Phase 0 docs; locate the argument-hint definition (the literal argument-hint: "<scope: ...>") in .claude/skills/codebase-audit/SKILL.md and add "src/synthorg/" to the scope enumeration (e.g. include "src/synthorg/" alongside existing scopes) so the help output and documentation are consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.claude/skills/codebase-audit/SKILL.md:
- Line 95: Remove the duplicate agent definition named "security-gaps" from the
Python backend section and keep only the cross-cutting definition; specifically,
delete the redundant `security-gaps` entry in the Python backend block (also
duplicated later) so there is a single canonical agent definition under the
cross-cutting section and ensure any references or scheduling config point to
that one symbol.
- Around line 119-125: The infra agent mappings are ambiguous: ensure
`docker-infra` only targets the `docker/` scope and `ci-workflows` only targets
the `.github/` scope by changing the table rows so `docker-infra` lists only
"Dockerfile issues, compose config, port security, healthchecks" with scope
`docker/`, and `ci-workflows` lists only "Missing timeouts, script injection,
permissions gaps, silent failures" with scope `.github/`; also update the
repeated wording at lines referencing 34-35 to match this scope-specific mapping
to prevent cross-run of agents.
---
Duplicate comments:
In @.claude/skills/codebase-audit/SKILL.md:
- Line 3: Update the argument-hint string to include the documented
src/synthorg/ scope so help text matches Phase 0 docs; locate the argument-hint
definition (the literal argument-hint: "<scope: ...>") in
.claude/skills/codebase-audit/SKILL.md and add "src/synthorg/" to the scope
enumeration (e.g. include "src/synthorg/" alongside existing scopes) so the help
output and documentation are consistent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: fea5077c-4b93-414c-b237-cd2bdc5dbdb2
📒 Files selected for processing (1)
.claude/skills/codebase-audit/SKILL.md
- Add src/synthorg/ to argument-hint for consistency with Phase 0 docs - Remove duplicate security-gaps from Python Backend (keep cross-cutting only) - Clarify infra agent scoping: docker-infra for docker/, ci-workflows for .github/ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.claude/skills/codebase-audit/SKILL.md:
- Around line 172-175: Replace the "Launch ALL selected agents in parallel"
behavior by implementing a bounded worker pool that enqueues selected agents and
dispatches at most a configurable concurrency (suggest 6-10) to the Agent tool
using run_in_background: true; update the launching logic that currently emits
"Launched N audit agents in parallel. Waiting for results..." to instead report
how many were started immediately, how many are queued, and overall total, and
add retry/backoff for transient failures (exponential backoff with a few
retries) when invoking the Agent tool; look for the launch/dispatch code that
constructs Agent tool calls (references: run_in_background: true and the
"Launched N audit agents in parallel. Waiting for results..." message) and
replace the fire-and-forget loop with a bounded worker queue, concurrency
limiter, and retry/backoff.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 20888417-c379-4343-9809-4b3f76f5c184
📒 Files selected for processing (1)
.claude/skills/codebase-audit/SKILL.md
| Launch ALL selected agents in parallel using the Agent tool with `run_in_background: true`. Give each a descriptive `name` for tracking. | ||
|
|
||
| Track agent count and report to user: "Launched N audit agents in parallel. Waiting for results..." | ||
|
|
There was a problem hiding this comment.
Bound parallelism instead of launching all agents at once.
Line 172’s “Launch ALL selected agents in parallel” can overload local resources and hit GitHub/API rate limits, causing partial or failed audits. Add a bounded worker pool (for example, max 6-10 concurrent agents) with queueing and retry/backoff.
Suggested update
-Launch ALL selected agents in parallel using the Agent tool with `run_in_background: true`. Give each a descriptive `name` for tracking.
+Launch selected agents using bounded parallelism (worker pool) with `run_in_background: true`.
+Recommended: max 6-10 concurrent agents, queue the rest, and apply retry/backoff on transient failures.
+Give each agent a descriptive `name` for tracking.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.claude/skills/codebase-audit/SKILL.md around lines 172 - 175, Replace the
"Launch ALL selected agents in parallel" behavior by implementing a bounded
worker pool that enqueues selected agents and dispatches at most a configurable
concurrency (suggest 6-10) to the Agent tool using run_in_background: true;
update the launching logic that currently emits "Launched N audit agents in
parallel. Waiting for results..." to instead report how many were started
immediately, how many are queued, and overall total, and add retry/backoff for
transient failures (exponential backoff with a few retries) when invoking the
Agent tool; look for the launch/dispatch code that constructs Agent tool calls
(references: run_in_background: true and the "Launched N audit agents in
parallel. Waiting for results..." message) and replace the fire-and-forget loop
with a bounded worker queue, concurrency limiter, and retry/backoff.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
🤖 I have created a release *beep* *boop* --- ## [0.3.8](v0.3.7...v0.3.8) (2026-03-19) ### Features * dynamic settings UI with auto-discovery and basic/advanced toggle ([#600](#600)) ([11b32b7](11b32b7)) ### Bug Fixes * **ci:** remove dst: . from GoReleaser archive config ([#598](#598)) ([c8bf862](c8bf862)) * **engine:** wire compaction_callback and plan_execute_config through auto-selection ([#601](#601)) ([795327f](795327f)) * harden setup wizard completion and status checks ([#616](#616)) ([d99d7b7](d99d7b7)) ### Maintenance * add /codebase-audit skill for deep parallel codebase auditing ([#613](#613)) ([db02320](db02320)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Summary
/codebase-auditslash command -- a reusable Claude Code skill that orchestrates 20+ specialized parallel agents to audit the entire codebase (or targeted scope) for issuesKey Features
full,src/,web/,cli/,docs/,.github/--report-onlyto skip issue creationSkill Phases
Test plan
/codebase-audit fulland verify all phases execute/codebase-audit src/synthorg/api/for targeted scope/codebase-audit --report-onlyand verify no issues createdSummary by CodeRabbit