fix: correct stale conformance counts in SKILL.md and AGENTS.md; extend check_doc_counts.py#423
Conversation
…nd check_doc_counts.py SKILL.md said 59 conformance programs; AGENTS.md said 62 (×3) and 28 examples (×2). All were correct at the time of writing but drifted as new conformance programs were added (65→67→70). check_doc_counts.py only validated TESTING.md, CONTRIBUTING.md, CLAUDE.md, and README.md — SKILL.md and AGENTS.md were never added to its coverage. - SKILL.md: 59 → 70 - AGENTS.md: 62 → 70 (×3), 28 → 30 (×2) - scripts/check_doc_counts.py: add sections 10 (SKILL.md) and 11 (AGENTS.md) so both files are validated on every pre-commit run Caught by independent assessment of v0.0.103. Co-Authored-By: Claude <noreply@anthropic.invalid>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughDocumentation counts updated to reflect an expanded conformance corpus (70 programs) and examples (30). Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #423 +/- ##
=======================================
Coverage 90.30% 90.30%
=======================================
Files 49 49
Lines 19163 19163
Branches 220 220
=======================================
Hits 17306 17306
Misses 1853 1853
Partials 4 4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…nts.py to FAQ and ROADMAP FAQ.md had two stale conformance counts (55 and 65 → 70). Extends check_doc_counts.py sections 12 (FAQ.md) and 13 (ROADMAP.md "Where we are" summary) so all major public-facing docs are now validated on every pre-commit run. Co-Authored-By: Claude <noreply@anthropic.invalid>
Co-Authored-By: Claude <noreply@anthropic.invalid>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/check_doc_counts.py`:
- Around line 447-462: The ROADMAP.md search currently uses re.search over the
whole file (roadmap_md) and can match the wrong occurrence; update the logic to
first extract the specific "Where we are" section from roadmap_md (e.g., via a
section-header regex), then run the count regex with re.DOTALL to allow
newlines, and replace the existing re.search(...) usage with this scoped search;
also add explicit failure handling if the "Where we are" section is not found or
if the count pattern does not match (raise or append an error instead of
silently continuing) so that doc_tests, doc_conf, and the comparisons against
live_total_tests/live_conformance always operate on the intended text.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: f8bfdfe8-27cb-4582-bce0-670810cb677c
⛔ Files ignored due to path filters (1)
docs/llms-full.txtis excluded by!docs/**
📒 Files selected for processing (2)
FAQ.mdscripts/check_doc_counts.py
ROADMAP.md: update independent assessment rating from 60-70% to 65-75% (March 2026 assessment), with updated characterisation of the remaining gap and explicit pointer to #225 as the most important next step. ci.yml: add "Check documentation counts are consistent" step running check_doc_counts.py in the test job. Previously only ran in pre-commit hooks — PRs from forks or direct API pushes could bypass it. Co-Authored-By: Claude <noreply@anthropic.invalid>
There was a problem hiding this comment.
♻️ Duplicate comments (1)
scripts/check_doc_counts.py (1)
449-468:⚠️ Potential issue | 🟠 MajorScope the ROADMAP.md check to the "Where we are" section.
The regex on line 455 operates on the entire file, and without
re.DOTALL, the.*?cannot span newlines. ROADMAP.md contains multiple occurrences of test/conformance count patterns (e.g., in v0.0.102, v0.0.101 changelog entries and the final summary line), so an unscoped match can target the wrong occurrence. If the "Where we are" section is ever reflowed across multiple lines, the pattern will also silently fail to match.Extract the section first, then search with
re.DOTALL:🔧 Suggested hardening
roadmap_md = (root / "ROADMAP.md").read_text() - m = re.search(r"([\d,]+) tests,.*?(\d+) conformance programs", roadmap_md) - if m: - doc_tests = int(m.group(1).replace(",", "")) - doc_conf = int(m.group(2)) - if doc_tests != live_total_tests: - errors.append( - f"ROADMAP.md: test count: doc says {doc_tests}," - f" live is {live_total_tests}" - ) - if doc_conf != live_conformance: - errors.append( - f"ROADMAP.md: conformance count: doc says {doc_conf}," - f" live is {live_conformance}" - ) + section_match = re.search( + r"^## Where we are\b(.*?)(?=^##\s|\Z)", + roadmap_md, + re.MULTILINE | re.DOTALL, + ) + if not section_match: + errors.append('ROADMAP.md: could not find "Where we are" section') + else: + m = re.search( + r"([\d,]+) tests,.*?(\d+) conformance programs", + section_match.group(1), + re.DOTALL, + ) + if not m: + errors.append( + 'ROADMAP.md: could not find test/conformance counts in ' + '"Where we are" section' + ) + else: + doc_tests = int(m.group(1).replace(",", "")) + doc_conf = int(m.group(2)) + if doc_tests != live_total_tests: + errors.append( + f"ROADMAP.md: test count: doc says {doc_tests}," + f" live is {live_total_tests}" + ) + if doc_conf != live_conformance: + errors.append( + f"ROADMAP.md: conformance count: doc says {doc_conf}," + f" live is {live_conformance}" + )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/check_doc_counts.py` around lines 449 - 468, The ROADMAP.md regex currently scans the whole file via roadmap_md and may match the wrong occurrence or fail across newlines; change the logic to first extract the "Where we are" section (e.g., locate the header text "## Where we are" or a unique surrounding marker) from roadmap_md into a smaller string (section_text) and then run the existing search using re.search with re.DOTALL (or re.S) against section_text so the pattern can span newlines and only matches the summary; update references to m, doc_tests and doc_conf accordingly to use the match from the scoped section.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@scripts/check_doc_counts.py`:
- Around line 449-468: The ROADMAP.md regex currently scans the whole file via
roadmap_md and may match the wrong occurrence or fail across newlines; change
the logic to first extract the "Where we are" section (e.g., locate the header
text "## Where we are" or a unique surrounding marker) from roadmap_md into a
smaller string (section_text) and then run the existing search using re.search
with re.DOTALL (or re.S) against section_text so the pattern can span newlines
and only matches the summary; update references to m, doc_tests and doc_conf
accordingly to use the match from the scoped section.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: b7f0e51a-c284-47cb-a7ae-a12e99cb2054
📒 Files selected for processing (1)
scripts/check_doc_counts.py
…losed #378 - #413 description updated to reflect provider registry refactor scope - #425 (xAI Grok, blocked by #413) added to Phase 2a inference hardening - #378 (Inference timeout, closed in v0.0.103) removed from Phase 2a list and quick-wins table per no-stale-roadmap-items policy Co-Authored-By: Claude <noreply@anthropic.invalid>
Previously used re.search over the full file, which matched the first occurrence of the count pattern — correct by accident (newest-first ordering) but fragile. ROADMAP.md contains historical per-release snapshots with older counts that would match instead if ordering changed. Now extracts the '## Where we are' section first, then searches within it. Also adds explicit errors if the section or pattern is absent, rather than silently skipping. Raised by CodeRabbit on PR #423. Co-Authored-By: Claude <noreply@anthropic.invalid>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@ROADMAP.md`:
- Line 69: Update the documentation and any counts to reflect that there are
four OpenAI-compatible providers (openai, moonshot, mistral, xAI) instead of
three: adjust the description around the `_call_inference_provider()` refactor
to state four structurally identical OpenAI-compatible endpoints, and ensure
references to the `_ProviderConfig` dataclass and `_PROVIDERS` registry note
that Anthropic remains the only non-compatible provider; keep the explanation
that adding further providers is a one-row change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 593203d9-efb6-4920-a95b-7d193aceedc3
📒 Files selected for processing (1)
ROADMAP.md
Summary
Fixes stale conformance and example counts in
SKILL.mdandAGENTS.md, and extendscheck_doc_counts.pyto cover both files so this can't drift silently again.What was wrong
check_doc_counts.pyvalidated TESTING.md, CONTRIBUTING.md, CLAUDE.md, and README.md — butSKILL.mdandAGENTS.mdwere never added. Counts drifted as conformance programs were added (65→67→70) without those files being updated.SKILL.mdconformance countAGENTS.mdconformance count (×3)AGENTS.mdexample count (×2)What was fixed
check_doc_counts.pycoveringSKILL.mdandAGENTS.md— any future drift will be caught at pre-commit timeCaught by independent assessment of v0.0.103.
🤖 Generated with Claude Code
Summary by CodeRabbit
Documentation
Chores