Skip to content

fix(ci): enforce catalog count integrity#525

Merged
affaan-m merged 2 commits into
affaan-m:mainfrom
justinphilpott:fix/catalog-integrity-validation
Mar 16, 2026
Merged

fix(ci): enforce catalog count integrity#525
affaan-m merged 2 commits into
affaan-m:mainfrom
justinphilpott:fix/catalog-integrity-validation

Conversation

@justinphilpott

@justinphilpott justinphilpott commented Mar 16, 2026

Copy link
Copy Markdown
Contributor

Description

Fix catalog count drift by updating stale public counts and enforcing the existing catalog validator in normal test/CI paths.

Related context: #456, #438
Follow-up for localized docs: #527

Type of Change

  • fix: Bug fix
  • test: Tests
  • ci: CI/CD changes

Summary

  • update stale agent, command, and skill counts in README.md and AGENTS.md
  • add scripts/ci/catalog.js --text to the standard test and CI validate paths
  • extend catalog validation to cover AGENTS.md project structure entries
  • add regression tests for real counts, drift cases, and formatting variations

Testing

  • pnpm test

Checklist

  • Tests pass locally
  • Validation scripts pass
  • Follows conventional commits format
  • Updated relevant documentation

Summary by CodeRabbit

  • Documentation
    • Updated README and AGENTS.md counts and project-structure listings (Agents 21, Skills 102, Commands 52).
  • Chores
    • Added catalog validation to the CI pipeline to check documentation counts.
    • Extended the test suite with catalog validation fixtures and tests to detect and report documentation drift.

@coderabbitai

coderabbitai Bot commented Mar 16, 2026

Copy link
Copy Markdown
Contributor

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5b1b4293-2ff3-466e-9a5b-a4e1b3332b9d

📥 Commits

Reviewing files that changed from the base of the PR and between dca7c10 and 3fe0389.

📒 Files selected for processing (6)
  • .github/workflows/ci.yml
  • AGENTS.md
  • README.md
  • package.json
  • scripts/ci/catalog.js
  • tests/ci/validators.test.js

📝 Walkthrough

Walkthrough

Adds a catalog-count validator to CI and tests; CI workflow and npm test run node scripts/ci/catalog.js --text. Updates documentation counts in README and AGENTS and extends catalog parsing to validate project-structure lines; new tests exercise the validator.

Changes

Cohort / File(s) Summary
CI Integration
.github/workflows/ci.yml, package.json
Added "Validate catalog counts" step to CI (runs node scripts/ci/catalog.js --text) and inserted catalog validation into the npm test script before running tests.
Documentation Updates
README.md, AGENTS.md
Updated static counts: README agents 16→21, skills 65→102, commands 40→52; AGENTS.md counts updated (agents 20→21, skills 65+→102, commands 51→52) and project-structure listing adjusted.
Catalog Validator
scripts/ci/catalog.js
Enhanced parsing to accept optional bolded table headers; added project-structure validation for AGENTS.md producing exact/minimum expectations, and merged these into returned expectations; throws if required structure entries are missing.
Validator Tests
tests/ci/validators.test.js
Added test helpers (runCatalogValidator, writeCatalogFixture) and comprehensive tests for catalog.js covering passing, drift-detection, and varied formatting/spacing in structure lines.

Sequence Diagram(s)

sequenceDiagram
    participant CI as CI Pipeline
    participant Catalog as Catalog Validator
    participant Files as README/AGENTS
    participant Repo as Repository (agents/, skills/, commands/)
    participant Report as Validation Report

    CI->>Catalog: run catalog.js --text
    Catalog->>Files: read README.md & AGENTS.md
    Files-->>Catalog: return documented counts & structure lines
    Catalog->>Repo: enumerate actual files/directories
    Repo-->>Catalog: return actual counts
    Catalog->>Catalog: build expectations (tables + project-structure)
    Catalog->>Catalog: compare expected vs actual
    alt match
        Catalog->>Report: success message
    else mismatch
        Catalog->>Report: failure details
        Report-->>CI: mark validation failed
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Poem

🐰
Counts in README and AGENTS align,
A tiny validator hops down the line.
I tally agents, skills, and commands with cheer,
Keeping docs and repo counts close and clear.
—thump, a rabbit clap for CI gear 🥕

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding catalog count validation to enforce data integrity in CI.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Tip

You can disable the changed files summary in the walkthrough.

Disable the reviews.changed_files_summary setting to disable the changed files summary in the walkthrough.

@justinphilpott

Copy link
Copy Markdown
Contributor Author

Followed up on the review notes with one additional commit:

  • hardened catalog.js AGENTS structure parsing to tolerate extra whitespace and - / / separators
  • added a regression test covering those formatting variations
  • opened translation drift follow-up issue #527 so the localized-doc scope exclusion is tracked explicitly

Validation rerun: pnpm test OK

@justinphilpott justinphilpott marked this pull request as ready for review March 16, 2026 18:33
@greptile-apps

greptile-apps Bot commented Mar 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes catalog count drift by bumping stale documentation counts (agents: 16→21, skills: 65→102, commands: 40→52 in README.md; 20/65+/51→21/102/52 in AGENTS.md) and enforcing the existing catalog.js validator in both the pnpm test script and the CI validate job. The structure-section parsing added to parseAgentsDocExpectations is a meaningful new enforcement layer, and the regression tests give good baseline coverage.

Key observations:

  • The second README.md comparison table (| **Agents** | 21 |) uses plain numbers without and is not matched by any tablePatterns regex — not the old one and not the new one with (?:\*\*)?. This table's counts will not be caught by the CI check if they drift in the future. The (?:\*\*)? addition appears targeted at this table but doesn't reach it because the requirement also needs to be relaxed or a separate pattern added.
  • The skills structure-section mode ternary is a no-op (always resolves to 'minimum'), meaning a skills count lower than documented will not be caught via the structure block — this was flagged in a prior review thread.
  • The test helper patches catalog.js source via a .*? regex, which is brittle to multi-line declarations — also previously flagged.

Confidence Score: 3/5

  • Safe to merge for documentation and pipeline wiring; the logic gaps leave some count-drift paths undetected.
  • The count updates are correct and confirmed by counting actual filesystem entries (21 agents, 52 commands, 102 skills). The CI wiring is straightforward. However, two gaps reduce confidence: (1) the second README comparison table is updated but not validated — a future drift there would pass CI silently; (2) the skills structure-section mode bug (always 'minimum', previously flagged) means a reduction in skill count goes undetected through that check path. These are functional blind spots in the enforcement mechanism that this PR introduces.
  • scripts/ci/catalog.js (tablePatterns regex coverage) and tests/ci/validators.test.js (README fixture completeness and structure-section isolation)

Important Files Changed

Filename Overview
scripts/ci/catalog.js Extends parseAgentsDocExpectations to also validate the AGENTS.md project structure block, and updates tablePatterns regexes to tolerate bold markdown markers. The skills structure check mode is hardcoded to 'minimum' regardless of whether '+' is written, and the second README comparison table (no ✅) remains unvalidated.
tests/ci/validators.test.js Adds runCatalogValidator and writeCatalogFixture helpers plus three catalog tests (real counts, drift, formatting). Constant patching via regex and the single-table README fixture are brittle; the drift test doesn't isolate the structure-section path specifically. These issues were flagged in prior review threads.
.github/workflows/ci.yml Adds a "Validate catalog counts" step running catalog.js --text with continue-on-error: false, correctly placed after the existing validate-rules step.
package.json Inserts catalog.js --text into the test script between validate-no-personal-paths and run-all.js, consistent with the CI pipeline order.
README.md Updates four count occurrences from stale values (16/65/40) to current counts (21/102/52); changes are consistent across all tables. The second comparison table's format (bold headers, no ✅) remains outside catalog validation coverage.
AGENTS.md Updates the summary line and project structure block to match current actual counts (21 agents / 102 skills / 52 commands). Removed the '+' from the old '65+ skills', making skills validation mode exact in the summary section.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[catalog.js --text] --> B[buildCatalog\nCount agents, commands, skills\nfrom filesystem]
    A --> C[parseReadmeExpectations\nREADME.md]
    A --> D[parseAgentsDocExpectations\nAGENTS.md]

    C --> C1[Quick-start line\naccess to N agents M skills P commands\nmode: exact x3]
    C --> C2[First table rows with checkmark\nAgents N agents row\nmode: exact x3]
    C --> C3[Second bold table rows\nAgents N row - no checkmark\nNOT matched by any pattern]

    D --> D1[Summary line\nproviding N specialized agents M skills P commands\nmode: exact or minimum for skills]
    D --> D2[Structure block lines\nagents slash N subagents\nskills slash N workflow skills\ncommands slash N slash commands\nWARN: skills always minimum mode]

    B --> E[evaluateExpectations\n12 total checks]
    C1 --> E
    C2 --> E
    D1 --> E
    D2 --> E

    E --> F{Any check fails?}
    F -- yes --> G[Print mismatches\nprocess.exit 1]
    F -- no --> H[Documentation counts match\nprocess.exit 0]
Loading

Last reviewed commit: 3fe0389

Comment thread scripts/ci/catalog.js
Comment thread tests/ci/validators.test.js
Comment thread tests/ci/validators.test.js

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
tests/ci/validators.test.js (1)

361-379: Add one regression test for skills exactness when + is absent.

You already test spacing/dash variants; please add a case that proves non-+ skills structure entries are treated as exact (not minimum). This will lock in the intended integrity behavior.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/ci/validators.test.js` around lines 361 - 379, Add a regression test
that verifies "skills" entries without a '+' are enforced as exact counts:
create a new test (using test(), createTestDir(), writeCatalogFixture(),
runCatalogValidator(), cleanupTestDir()) that writes a README fixture with a
skills line like 'skills/ - 1 workflow skills and domain knowledge' (no '+') but
populates the agents/skills folder with a different number (e.g., 2) and then
assert runCatalogValidator returns a non-zero code and stderr indicates an
exact-count mismatch; use assert.strictEqual to check the code and include
cleanupTestDir at the end.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@scripts/ci/catalog.js`:
- Around line 147-149: The mode expression for skill patterns is inverted
causing it to always set 'minimum'; change the conditional in the object literal
that assigns mode so that when pattern.mode === 'minimum' and there is NO '+'
(i.e. match[2] is falsy) it becomes 'exact' rather than 'minimum'. In other
words, update the mode assignment near where mode, expected, source are set (the
object using pattern.mode and match) to use pattern.mode === 'minimum' &&
!match[2] ? 'exact' : pattern.mode.

---

Nitpick comments:
In `@tests/ci/validators.test.js`:
- Around line 361-379: Add a regression test that verifies "skills" entries
without a '+' are enforced as exact counts: create a new test (using test(),
createTestDir(), writeCatalogFixture(), runCatalogValidator(), cleanupTestDir())
that writes a README fixture with a skills line like 'skills/ - 1 workflow
skills and domain knowledge' (no '+') but populates the agents/skills folder
with a different number (e.g., 2) and then assert runCatalogValidator returns a
non-zero code and stderr indicates an exact-count mismatch; use
assert.strictEqual to check the code and include cleanupTestDir at the end.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 392b4346-5a13-4856-b1ff-2c3c567aafad

📥 Commits

Reviewing files that changed from the base of the PR and between b489309 and dca7c10.

📒 Files selected for processing (6)
  • .github/workflows/ci.yml
  • AGENTS.md
  • README.md
  • package.json
  • scripts/ci/catalog.js
  • tests/ci/validators.test.js

Comment thread scripts/ci/catalog.js
Comment on lines +147 to +149
mode: pattern.mode === 'minimum' && match[2] ? 'minimum' : pattern.mode,
expected: Number(match[1]),
source: `${pattern.source} (${pattern.category})`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Skills project-structure checks currently never become exact.

For the skills pattern, this expression always resolves to minimum, even when no + is present. That can let count drift pass when the structure row is intended to be exact.

💡 Proposed fix
     expectations.push({
       category: pattern.category,
-      mode: pattern.mode === 'minimum' && match[2] ? 'minimum' : pattern.mode,
+      mode: pattern.mode === 'minimum'
+        ? (match[2] ? 'minimum' : 'exact')
+        : pattern.mode,
       expected: Number(match[1]),
       source: `${pattern.source} (${pattern.category})`
     });
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/ci/catalog.js` around lines 147 - 149, The mode expression for skill
patterns is inverted causing it to always set 'minimum'; change the conditional
in the object literal that assigns mode so that when pattern.mode === 'minimum'
and there is NO '+' (i.e. match[2] is falsy) it becomes 'exact' rather than
'minimum'. In other words, update the mode assignment near where mode, expected,
source are set (the object using pattern.mode and match) to use pattern.mode ===
'minimum' && !match[2] ? 'exact' : pattern.mode.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 6 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="AGENTS.md">

<violation number="1" location="AGENTS.md:3">
P2: AGENTS.md now claims 18 agents, but the Available Agents table still lists only 16 entries and omits the Kotlin agents present in the catalog. This makes the documentation internally inconsistent and hides existing agents.</violation>
</file>

<file name="scripts/ci/catalog.js">

<violation number="1" location="scripts/ci/catalog.js:127">
P2: Skills count in AGENTS.md project structure is always treated as a minimum because the skills structure pattern is hard-coded to `mode: 'minimum'`, so even without a `+` the check never enforces exact counts.</violation>
</file>

Since this is your first cubic review, here's how it works:

  • cubic automatically reviews your code and comments on bugs and improvements
  • Teach cubic by replying to its comments. cubic learns from your replies and gets better over time
  • Add one-off context when rerunning by tagging @cubic-dev-ai with guidance or docs links (including llms.txt)
  • Ask questions if you need clarification on any suggestion

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread AGENTS.md
@@ -1,6 +1,6 @@
# Everything Claude Code (ECC) — Agent Instructions

@cubic-dev-ai cubic-dev-ai Bot Mar 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: AGENTS.md now claims 18 agents, but the Available Agents table still lists only 16 entries and omits the Kotlin agents present in the catalog. This makes the documentation internally inconsistent and hides existing agents.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At AGENTS.md, line 3:

<comment>AGENTS.md now claims 18 agents, but the Available Agents table still lists only 16 entries and omits the Kotlin agents present in the catalog. This makes the documentation internally inconsistent and hides existing agents.</comment>

<file context>
@@ -1,6 +1,6 @@
 # Everything Claude Code (ECC) — Agent Instructions
 
-This is a **production-ready AI coding plugin** providing 16 specialized agents, 65+ skills, 40 commands, and automated hook workflows for software development.
+This is a **production-ready AI coding plugin** providing 18 specialized agents, 94 skills, 48 commands, and automated hook workflows for software development.
 
 ## Core Principles
</file context>
Fix with Cubic

Comment thread scripts/ci/catalog.js
@affaan-m affaan-m force-pushed the fix/catalog-integrity-validation branch from dca7c10 to 3fe0389 Compare March 16, 2026 20:37
@affaan-m affaan-m merged commit 01ed1b3 into affaan-m:main Mar 16, 2026
2 of 3 checks passed
Comment thread scripts/ci/catalog.js
Comment on lines 78 to 82
const tablePatterns = [
{ category: 'agents', regex: /\|\s*Agents\s*\|\s*✅\s*(\d+)\s+agents\s*\|/i, source: 'README.md comparison table' },
{ category: 'commands', regex: /\|\s*Commands\s*\|\s*✅\s*(\d+)\s+commands\s*\|/i, source: 'README.md comparison table' },
{ category: 'skills', regex: /\|\s*Skills\s*\|\s*✅\s*(\d+)\s+skills\s*\|/i, source: 'README.md comparison table' }
{ category: 'agents', regex: /\|\s*(?:\*\*)?Agents(?:\*\*)?\s*\|\s*✅\s*(\d+)\s+agents\s*\|/i, source: 'README.md comparison table' },
{ category: 'commands', regex: /\|\s*(?:\*\*)?Commands(?:\*\*)?\s*\|\s*✅\s*(\d+)\s+commands\s*\|/i, source: 'README.md comparison table' },
{ category: 'skills', regex: /\|\s*(?:\*\*)?Skills(?:\*\*)?\s*\|\s*✅\s*(\d+)\s+skills\s*\|/i, source: 'README.md comparison table' }
];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second comparison table rows not covered by validation

The tablePatterns regexes require ✅\s*(\d+)\s+agents etc., which matches the first comparison table (| Agents | ✅ 21 agents | ✅ 12 agents |). However, the second comparison table in README.md uses bold headers and plain counts without — e.g. | **Agents** | 21 | Shared (AGENTS.md) |. This second table is updated in this PR but is never validated.

The (?:\*\*)? addition in this PR looks like it was intended to capture the bold-header rows, but since the second table also lacks ✅ N agents, the new regex still doesn't match it. If the second table's counts drift in the future, the CI check will pass silently.

Either extend the regex to match both formats, or add a dedicated pattern for the plain-count variant:

// Example: validate the plain-count table column separately
{ category: 'agents', regex: /\|\s*\*\*Agents\*\*\s*\|\s*(\d+)\s*\|/, source: 'README.md feature table (bold)' },

Comment on lines +199 to +200
fs.writeFileSync(readmePath, `Access to ${readmeCounts.agents} agents, ${readmeCounts.skills} skills, and ${readmeCounts.commands} commands.\n| Feature | Claude Code | Cursor IDE | Codex CLI | OpenCode |\n|---------|------------|------------|-----------|----------|\n| Agents | ✅ ${readmeCounts.agents} agents | Shared | Shared | 1 |\n| Commands | ✅ ${readmeCounts.commands} commands | Shared | Shared | 1 |\n| Skills | ✅ ${readmeCounts.skills} skills | Shared | Shared | 1 |\n`);
fs.writeFileSync(agentsPath, `This is a **production-ready AI coding plugin** providing ${summaryCounts.agents} specialized agents, ${summaryCounts.skills} skills, ${summaryCounts.commands} commands, and automated hook workflows for software development.\n\n\`\`\`\n${structureLines.join('\n')}\n\`\`\`\n`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README fixture missing the second comparison table

writeCatalogFixture writes only one table in the README fixture — the ✅-format table. The real README.md contains two tables with count rows (one with | Agents | ✅ 21 agents | and another with | **Agents** | 21 |). Since the catalog validator calls .match() on the full README content and returns the first match, the fixture's single-table approach validates the correct regex branch — but it also means no test confirms behavior when both tables are present and one of them drifts. A fixture that includes both table formats would give stronger coverage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants