fix(ci): enforce catalog count integrity#525
Conversation
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughAdds a catalog-count validator to CI and tests; CI workflow and npm test run Changes
Sequence Diagram(s)sequenceDiagram
participant CI as CI Pipeline
participant Catalog as Catalog Validator
participant Files as README/AGENTS
participant Repo as Repository (agents/, skills/, commands/)
participant Report as Validation Report
CI->>Catalog: run catalog.js --text
Catalog->>Files: read README.md & AGENTS.md
Files-->>Catalog: return documented counts & structure lines
Catalog->>Repo: enumerate actual files/directories
Repo-->>Catalog: return actual counts
Catalog->>Catalog: build expectations (tables + project-structure)
Catalog->>Catalog: compare expected vs actual
alt match
Catalog->>Report: success message
else mismatch
Catalog->>Report: failure details
Report-->>CI: mark validation failed
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Tip You can disable the changed files summary in the walkthrough.Disable the |
|
Followed up on the review notes with one additional commit:
Validation rerun: |
Greptile SummaryThis PR fixes catalog count drift by bumping stale documentation counts (agents: 16→21, skills: 65→102, commands: 40→52 in Key observations:
Confidence Score: 3/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[catalog.js --text] --> B[buildCatalog\nCount agents, commands, skills\nfrom filesystem]
A --> C[parseReadmeExpectations\nREADME.md]
A --> D[parseAgentsDocExpectations\nAGENTS.md]
C --> C1[Quick-start line\naccess to N agents M skills P commands\nmode: exact x3]
C --> C2[First table rows with checkmark\nAgents N agents row\nmode: exact x3]
C --> C3[Second bold table rows\nAgents N row - no checkmark\nNOT matched by any pattern]
D --> D1[Summary line\nproviding N specialized agents M skills P commands\nmode: exact or minimum for skills]
D --> D2[Structure block lines\nagents slash N subagents\nskills slash N workflow skills\ncommands slash N slash commands\nWARN: skills always minimum mode]
B --> E[evaluateExpectations\n12 total checks]
C1 --> E
C2 --> E
D1 --> E
D2 --> E
E --> F{Any check fails?}
F -- yes --> G[Print mismatches\nprocess.exit 1]
F -- no --> H[Documentation counts match\nprocess.exit 0]
Last reviewed commit: 3fe0389 |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tests/ci/validators.test.js (1)
361-379: Add one regression test for skills exactness when+is absent.You already test spacing/dash variants; please add a case that proves non-
+skills structure entries are treated as exact (not minimum). This will lock in the intended integrity behavior.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/ci/validators.test.js` around lines 361 - 379, Add a regression test that verifies "skills" entries without a '+' are enforced as exact counts: create a new test (using test(), createTestDir(), writeCatalogFixture(), runCatalogValidator(), cleanupTestDir()) that writes a README fixture with a skills line like 'skills/ - 1 workflow skills and domain knowledge' (no '+') but populates the agents/skills folder with a different number (e.g., 2) and then assert runCatalogValidator returns a non-zero code and stderr indicates an exact-count mismatch; use assert.strictEqual to check the code and include cleanupTestDir at the end.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/ci/catalog.js`:
- Around line 147-149: The mode expression for skill patterns is inverted
causing it to always set 'minimum'; change the conditional in the object literal
that assigns mode so that when pattern.mode === 'minimum' and there is NO '+'
(i.e. match[2] is falsy) it becomes 'exact' rather than 'minimum'. In other
words, update the mode assignment near where mode, expected, source are set (the
object using pattern.mode and match) to use pattern.mode === 'minimum' &&
!match[2] ? 'exact' : pattern.mode.
---
Nitpick comments:
In `@tests/ci/validators.test.js`:
- Around line 361-379: Add a regression test that verifies "skills" entries
without a '+' are enforced as exact counts: create a new test (using test(),
createTestDir(), writeCatalogFixture(), runCatalogValidator(), cleanupTestDir())
that writes a README fixture with a skills line like 'skills/ - 1 workflow
skills and domain knowledge' (no '+') but populates the agents/skills folder
with a different number (e.g., 2) and then assert runCatalogValidator returns a
non-zero code and stderr indicates an exact-count mismatch; use
assert.strictEqual to check the code and include cleanupTestDir at the end.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 392b4346-5a13-4856-b1ff-2c3c567aafad
📒 Files selected for processing (6)
.github/workflows/ci.ymlAGENTS.mdREADME.mdpackage.jsonscripts/ci/catalog.jstests/ci/validators.test.js
| mode: pattern.mode === 'minimum' && match[2] ? 'minimum' : pattern.mode, | ||
| expected: Number(match[1]), | ||
| source: `${pattern.source} (${pattern.category})` |
There was a problem hiding this comment.
Skills project-structure checks currently never become exact.
For the skills pattern, this expression always resolves to minimum, even when no + is present. That can let count drift pass when the structure row is intended to be exact.
💡 Proposed fix
expectations.push({
category: pattern.category,
- mode: pattern.mode === 'minimum' && match[2] ? 'minimum' : pattern.mode,
+ mode: pattern.mode === 'minimum'
+ ? (match[2] ? 'minimum' : 'exact')
+ : pattern.mode,
expected: Number(match[1]),
source: `${pattern.source} (${pattern.category})`
});🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@scripts/ci/catalog.js` around lines 147 - 149, The mode expression for skill
patterns is inverted causing it to always set 'minimum'; change the conditional
in the object literal that assigns mode so that when pattern.mode === 'minimum'
and there is NO '+' (i.e. match[2] is falsy) it becomes 'exact' rather than
'minimum'. In other words, update the mode assignment near where mode, expected,
source are set (the object using pattern.mode and match) to use pattern.mode ===
'minimum' && !match[2] ? 'exact' : pattern.mode.
There was a problem hiding this comment.
2 issues found across 6 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="AGENTS.md">
<violation number="1" location="AGENTS.md:3">
P2: AGENTS.md now claims 18 agents, but the Available Agents table still lists only 16 entries and omits the Kotlin agents present in the catalog. This makes the documentation internally inconsistent and hides existing agents.</violation>
</file>
<file name="scripts/ci/catalog.js">
<violation number="1" location="scripts/ci/catalog.js:127">
P2: Skills count in AGENTS.md project structure is always treated as a minimum because the skills structure pattern is hard-coded to `mode: 'minimum'`, so even without a `+` the check never enforces exact counts.</violation>
</file>
Since this is your first cubic review, here's how it works:
- cubic automatically reviews your code and comments on bugs and improvements
- Teach cubic by replying to its comments. cubic learns from your replies and gets better over time
- Add one-off context when rerunning by tagging
@cubic-dev-aiwith guidance or docs links (includingllms.txt) - Ask questions if you need clarification on any suggestion
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| @@ -1,6 +1,6 @@ | |||
| # Everything Claude Code (ECC) — Agent Instructions | |||
There was a problem hiding this comment.
P2: AGENTS.md now claims 18 agents, but the Available Agents table still lists only 16 entries and omits the Kotlin agents present in the catalog. This makes the documentation internally inconsistent and hides existing agents.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At AGENTS.md, line 3:
<comment>AGENTS.md now claims 18 agents, but the Available Agents table still lists only 16 entries and omits the Kotlin agents present in the catalog. This makes the documentation internally inconsistent and hides existing agents.</comment>
<file context>
@@ -1,6 +1,6 @@
# Everything Claude Code (ECC) — Agent Instructions
-This is a **production-ready AI coding plugin** providing 16 specialized agents, 65+ skills, 40 commands, and automated hook workflows for software development.
+This is a **production-ready AI coding plugin** providing 18 specialized agents, 94 skills, 48 commands, and automated hook workflows for software development.
## Core Principles
</file context>
dca7c10 to
3fe0389
Compare
| const tablePatterns = [ | ||
| { category: 'agents', regex: /\|\s*Agents\s*\|\s*✅\s*(\d+)\s+agents\s*\|/i, source: 'README.md comparison table' }, | ||
| { category: 'commands', regex: /\|\s*Commands\s*\|\s*✅\s*(\d+)\s+commands\s*\|/i, source: 'README.md comparison table' }, | ||
| { category: 'skills', regex: /\|\s*Skills\s*\|\s*✅\s*(\d+)\s+skills\s*\|/i, source: 'README.md comparison table' } | ||
| { category: 'agents', regex: /\|\s*(?:\*\*)?Agents(?:\*\*)?\s*\|\s*✅\s*(\d+)\s+agents\s*\|/i, source: 'README.md comparison table' }, | ||
| { category: 'commands', regex: /\|\s*(?:\*\*)?Commands(?:\*\*)?\s*\|\s*✅\s*(\d+)\s+commands\s*\|/i, source: 'README.md comparison table' }, | ||
| { category: 'skills', regex: /\|\s*(?:\*\*)?Skills(?:\*\*)?\s*\|\s*✅\s*(\d+)\s+skills\s*\|/i, source: 'README.md comparison table' } | ||
| ]; |
There was a problem hiding this comment.
Second comparison table rows not covered by validation
The tablePatterns regexes require ✅\s*(\d+)\s+agents etc., which matches the first comparison table (| Agents | ✅ 21 agents | ✅ 12 agents |). However, the second comparison table in README.md uses bold headers and plain counts without ✅ — e.g. | **Agents** | 21 | Shared (AGENTS.md) |. This second table is updated in this PR but is never validated.
The (?:\*\*)? addition in this PR looks like it was intended to capture the bold-header rows, but since the second table also lacks ✅ N agents, the new regex still doesn't match it. If the second table's counts drift in the future, the CI check will pass silently.
Either extend the regex to match both formats, or add a dedicated pattern for the plain-count variant:
// Example: validate the plain-count table column separately
{ category: 'agents', regex: /\|\s*\*\*Agents\*\*\s*\|\s*(\d+)\s*\|/, source: 'README.md feature table (bold)' },| fs.writeFileSync(readmePath, `Access to ${readmeCounts.agents} agents, ${readmeCounts.skills} skills, and ${readmeCounts.commands} commands.\n| Feature | Claude Code | Cursor IDE | Codex CLI | OpenCode |\n|---------|------------|------------|-----------|----------|\n| Agents | ✅ ${readmeCounts.agents} agents | Shared | Shared | 1 |\n| Commands | ✅ ${readmeCounts.commands} commands | Shared | Shared | 1 |\n| Skills | ✅ ${readmeCounts.skills} skills | Shared | Shared | 1 |\n`); | ||
| fs.writeFileSync(agentsPath, `This is a **production-ready AI coding plugin** providing ${summaryCounts.agents} specialized agents, ${summaryCounts.skills} skills, ${summaryCounts.commands} commands, and automated hook workflows for software development.\n\n\`\`\`\n${structureLines.join('\n')}\n\`\`\`\n`); |
There was a problem hiding this comment.
README fixture missing the second comparison table
writeCatalogFixture writes only one table in the README fixture — the ✅-format table. The real README.md contains two tables with count rows (one with | Agents | ✅ 21 agents | and another with | **Agents** | 21 |). Since the catalog validator calls .match() on the full README content and returns the first match, the fixture's single-table approach validates the correct regex branch — but it also means no test confirms behavior when both tables are present and one of them drifts. A fixture that includes both table formats would give stronger coverage.
* fix(ci): enforce catalog count integrity * test: harden catalog structure parsing
Description
Fix catalog count drift by updating stale public counts and enforcing the existing catalog validator in normal test/CI paths.
Related context:
#456,#438Follow-up for localized docs:
#527Type of Change
fix:Bug fixtest:Testsci:CI/CD changesSummary
README.mdandAGENTS.mdscripts/ci/catalog.js --textto the standard test and CI validate pathsAGENTS.mdproject structure entriesTesting
pnpm testChecklist
Summary by CodeRabbit