Skip to content

fix: correct stale conformance counts in SKILL.md and AGENTS.md; extend check_doc_counts.py#423

Merged
aallan merged 6 commits into
mainfrom
fix/doc-count-drift
Mar 29, 2026
Merged

fix: correct stale conformance counts in SKILL.md and AGENTS.md; extend check_doc_counts.py#423
aallan merged 6 commits into
mainfrom
fix/doc-count-drift

Conversation

@aallan

@aallan aallan commented Mar 29, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes stale conformance and example counts in SKILL.md and AGENTS.md, and extends check_doc_counts.py to cover both files so this can't drift silently again.

What was wrong

check_doc_counts.py validated TESTING.md, CONTRIBUTING.md, CLAUDE.md, and README.md — but SKILL.md and AGENTS.md were never added. Counts drifted as conformance programs were added (65→67→70) without those files being updated.

File Was Should be
SKILL.md conformance count 59 70
AGENTS.md conformance count (×3) 62 70
AGENTS.md example count (×2) 28 30

What was fixed

  • Corrected all five stale numbers
  • Added sections 10 and 11 to check_doc_counts.py covering SKILL.md and AGENTS.md — any future drift will be caught at pre-commit time

Caught by independent assessment of v0.0.103.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation

    • Updated docs to reflect the expanded conformance test suite (now 70 programmes) and updated example count (now 30); FAQ and roadmap texts revised to match and clarify project status, viability assessment, and milestone wording.
  • Chores

    • Added broader automated checks to validate conformance/example counts across multiple docs and added a CI step to run those checks.

…nd check_doc_counts.py

SKILL.md said 59 conformance programs; AGENTS.md said 62 (×3) and 28
examples (×2). All were correct at the time of writing but drifted as
new conformance programs were added (65→67→70). check_doc_counts.py
only validated TESTING.md, CONTRIBUTING.md, CLAUDE.md, and README.md —
SKILL.md and AGENTS.md were never added to its coverage.

- SKILL.md: 59 → 70
- AGENTS.md: 62 → 70 (×3), 28 → 30 (×2)
- scripts/check_doc_counts.py: add sections 10 (SKILL.md) and 11
  (AGENTS.md) so both files are validated on every pre-commit run

Caught by independent assessment of v0.0.103.

Co-Authored-By: Claude <noreply@anthropic.invalid>
@coderabbitai

coderabbitai Bot commented Mar 29, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Documentation counts updated to reflect an expanded conformance corpus (70 programs) and examples (30). scripts/check_doc_counts.py was extended to parse and validate counts from SKILL.md, AGENTS.md, FAQ.md, and ROADMAP.md. CI now runs this validation script in the lint job.

Changes

Cohort / File(s) Summary
Docs: counts and wording
AGENTS.md, SKILL.md, FAQ.md, ROADMAP.md
Conformance counts updated to 70 across multiple docs; AGENTS.md examples count updated to 30; roadmap viability wording adjusted for v0.0.101 and milestone items rephrased.
Validation script
scripts/check_doc_counts.py
Significantly extended parsing and validation logic: extracts conformance/example totals from SKILL.md, multiple AGENTS.md sections, FAQ.md, and ROADMAP.md via regex; mismatches appended to errors and preserve non-zero exit behaviour.
Docs: minor
AGENTS.md, SKILL.md (inline examples counts)
Small numeric edits and comment updates to reflect new totals.
CI
.github/workflows/ci.yml
Added a lint-job step to run python scripts/check_doc_counts.py (placed after license check and before ruff).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested labels

ci, docs

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarises the primary changes: correcting stale conformance counts in documentation and extending the validation script to catch future drift.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/doc-count-drift

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov

codecov Bot commented Mar 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.30%. Comparing base (98fefaa) to head (37daa6e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #423   +/-   ##
=======================================
  Coverage   90.30%   90.30%           
=======================================
  Files          49       49           
  Lines       19163    19163           
  Branches      220      220           
=======================================
  Hits        17306    17306           
  Misses       1853     1853           
  Partials        4        4           
Flag Coverage Δ
javascript 50.56% <ø> (ø)
python 95.30% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

aallan and others added 2 commits March 29, 2026 13:30
…nts.py to FAQ and ROADMAP

FAQ.md had two stale conformance counts (55 and 65 → 70).
Extends check_doc_counts.py sections 12 (FAQ.md) and 13 (ROADMAP.md
"Where we are" summary) so all major public-facing docs are now
validated on every pre-commit run.

Co-Authored-By: Claude <noreply@anthropic.invalid>
Co-Authored-By: Claude <noreply@anthropic.invalid>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@scripts/check_doc_counts.py`:
- Around line 447-462: The ROADMAP.md search currently uses re.search over the
whole file (roadmap_md) and can match the wrong occurrence; update the logic to
first extract the specific "Where we are" section from roadmap_md (e.g., via a
section-header regex), then run the count regex with re.DOTALL to allow
newlines, and replace the existing re.search(...) usage with this scoped search;
also add explicit failure handling if the "Where we are" section is not found or
if the count pattern does not match (raise or append an error instead of
silently continuing) so that doc_tests, doc_conf, and the comparisons against
live_total_tests/live_conformance always operate on the intended text.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: f8bfdfe8-27cb-4582-bce0-670810cb677c

📥 Commits

Reviewing files that changed from the base of the PR and between 378b5d4 and 67dcb61.

⛔ Files ignored due to path filters (1)
  • docs/llms-full.txt is excluded by !docs/**
📒 Files selected for processing (2)
  • FAQ.md
  • scripts/check_doc_counts.py

Comment thread scripts/check_doc_counts.py
ROADMAP.md: update independent assessment rating from 60-70% to 65-75%
(March 2026 assessment), with updated characterisation of the remaining
gap and explicit pointer to #225 as the most important next step.

ci.yml: add "Check documentation counts are consistent" step running
check_doc_counts.py in the test job. Previously only ran in pre-commit
hooks — PRs from forks or direct API pushes could bypass it.

Co-Authored-By: Claude <noreply@anthropic.invalid>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
scripts/check_doc_counts.py (1)

449-468: ⚠️ Potential issue | 🟠 Major

Scope the ROADMAP.md check to the "Where we are" section.

The regex on line 455 operates on the entire file, and without re.DOTALL, the .*? cannot span newlines. ROADMAP.md contains multiple occurrences of test/conformance count patterns (e.g., in v0.0.102, v0.0.101 changelog entries and the final summary line), so an unscoped match can target the wrong occurrence. If the "Where we are" section is ever reflowed across multiple lines, the pattern will also silently fail to match.

Extract the section first, then search with re.DOTALL:

🔧 Suggested hardening
     roadmap_md = (root / "ROADMAP.md").read_text()
 
-    m = re.search(r"([\d,]+) tests,.*?(\d+) conformance programs", roadmap_md)
-    if m:
-        doc_tests = int(m.group(1).replace(",", ""))
-        doc_conf = int(m.group(2))
-        if doc_tests != live_total_tests:
-            errors.append(
-                f"ROADMAP.md: test count: doc says {doc_tests},"
-                f" live is {live_total_tests}"
-            )
-        if doc_conf != live_conformance:
-            errors.append(
-                f"ROADMAP.md: conformance count: doc says {doc_conf},"
-                f" live is {live_conformance}"
-            )
+    section_match = re.search(
+        r"^## Where we are\b(.*?)(?=^##\s|\Z)",
+        roadmap_md,
+        re.MULTILINE | re.DOTALL,
+    )
+    if not section_match:
+        errors.append('ROADMAP.md: could not find "Where we are" section')
+    else:
+        m = re.search(
+            r"([\d,]+) tests,.*?(\d+) conformance programs",
+            section_match.group(1),
+            re.DOTALL,
+        )
+        if not m:
+            errors.append(
+                'ROADMAP.md: could not find test/conformance counts in '
+                '"Where we are" section'
+            )
+        else:
+            doc_tests = int(m.group(1).replace(",", ""))
+            doc_conf = int(m.group(2))
+            if doc_tests != live_total_tests:
+                errors.append(
+                    f"ROADMAP.md: test count: doc says {doc_tests},"
+                    f" live is {live_total_tests}"
+                )
+            if doc_conf != live_conformance:
+                errors.append(
+                    f"ROADMAP.md: conformance count: doc says {doc_conf},"
+                    f" live is {live_conformance}"
+                )
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/check_doc_counts.py` around lines 449 - 468, The ROADMAP.md regex
currently scans the whole file via roadmap_md and may match the wrong occurrence
or fail across newlines; change the logic to first extract the "Where we are"
section (e.g., locate the header text "## Where we are" or a unique surrounding
marker) from roadmap_md into a smaller string (section_text) and then run the
existing search using re.search with re.DOTALL (or re.S) against section_text so
the pattern can span newlines and only matches the summary; update references to
m, doc_tests and doc_conf accordingly to use the match from the scoped section.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@scripts/check_doc_counts.py`:
- Around line 449-468: The ROADMAP.md regex currently scans the whole file via
roadmap_md and may match the wrong occurrence or fail across newlines; change
the logic to first extract the "Where we are" section (e.g., locate the header
text "## Where we are" or a unique surrounding marker) from roadmap_md into a
smaller string (section_text) and then run the existing search using re.search
with re.DOTALL (or re.S) against section_text so the pattern can span newlines
and only matches the summary; update references to m, doc_tests and doc_conf
accordingly to use the match from the scoped section.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: b7f0e51a-c284-47cb-a7ae-a12e99cb2054

📥 Commits

Reviewing files that changed from the base of the PR and between 67dcb61 and d4fd65d.

📒 Files selected for processing (1)
  • scripts/check_doc_counts.py

aallan and others added 2 commits March 29, 2026 13:44
…losed #378

- #413 description updated to reflect provider registry refactor scope
- #425 (xAI Grok, blocked by #413) added to Phase 2a inference hardening
- #378 (Inference timeout, closed in v0.0.103) removed from Phase 2a
  list and quick-wins table per no-stale-roadmap-items policy

Co-Authored-By: Claude <noreply@anthropic.invalid>
Previously used re.search over the full file, which matched the first
occurrence of the count pattern — correct by accident (newest-first
ordering) but fragile. ROADMAP.md contains historical per-release
snapshots with older counts that would match instead if ordering changed.

Now extracts the '## Where we are' section first, then searches within
it. Also adds explicit errors if the section or pattern is absent,
rather than silently skipping.

Raised by CodeRabbit on PR #423.

Co-Authored-By: Claude <noreply@anthropic.invalid>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@ROADMAP.md`:
- Line 69: Update the documentation and any counts to reflect that there are
four OpenAI-compatible providers (openai, moonshot, mistral, xAI) instead of
three: adjust the description around the `_call_inference_provider()` refactor
to state four structurally identical OpenAI-compatible endpoints, and ensure
references to the `_ProviderConfig` dataclass and `_PROVIDERS` registry note
that Anthropic remains the only non-compatible provider; keep the explanation
that adding further providers is a one-row change.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 593203d9-efb6-4920-a95b-7d193aceedc3

📥 Commits

Reviewing files that changed from the base of the PR and between bfecdfc and 604963d.

📒 Files selected for processing (1)
  • ROADMAP.md

Comment thread ROADMAP.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant