feat: validate vera run commands in examples/README.md (#361) by aallan · Pull Request #444 · aallan/vera

aallan · 2026-04-07T14:18:23Z

Summary

Adds scripts/check_examples_readme.py which parses every vera run command in the examples/README.md table and validates: (1) the referenced .vera file exists, (2) if --fn <name> is specified, public fn <name> appears in that file
Wires into pre-commit (triggers on examples/README.md or .vera file changes) and CI (runs alongside check_examples.py)
Updates hook count 22 → 23 in TESTING.md and CONTRIBUTING.md

All 28 current run commands pass.

Test plan

python scripts/check_examples_readme.py passes locally
Pre-commit hooks pass
CI will run the script on every PR touching examples/

Closes #361

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- Automated validation now checks example "vera run" commands in the examples README and fails if commands reference missing examples or non-public functions.
Chores
- Validation integrated into CI lint and pre-commit so it runs automatically on commits and in CI.
Documentation
- Contributor, testing and project docs, changelog and release metadata updated to list the new validation step and bump the project version.

Add scripts/check_examples_readme.py which: - Parses all table rows in examples/README.md - Extracts vera run commands from the Run column - Verifies each referenced .vera file exists - Verifies --fn <name> functions are actually public in the source Wire into pre-commit (triggered on examples/README.md or .vera changes) and CI (runs alongside check_examples.py). All 28 current commands pass. Closes #361 Co-Authored-By: Claude <noreply@anthropic.invalid>

coderabbitai · 2026-04-07T14:18:39Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: bff37ec9-1a2c-45f9-9006-39052b266fd0

📥 Commits

Reviewing files that changed from the base of the PR and between 52ebe08 and 4e19456.

⛔ Files ignored due to path filters (1)

docs/index.html is excluded by !docs/**

📒 Files selected for processing (1)

HISTORY.md

📝 Walkthrough

Walkthrough

Adds a new CLI script scripts/check_examples_readme.py and wires it into CI and pre-commit. The script parses backtick‑quoted vera run commands in examples/README.md, verifies referenced .vera files exist and that any --fn names are declared public fn in those files.

Changes

Cohort / File(s)	Summary
Validation Script `scripts/check_examples_readme.py`	New CLI that extracts backtick‑quoted `vera run ...` commands from `examples/README.md`, maps each to a `.vera` path and optional `--fn`, verifies file existence and presence of `public fn <name>`, prints failures and exits non‑zero on error.
CI / Pre-commit Hooks `.github/workflows/ci.yml`, `.pre-commit-config.yaml`	CI `lint` job and a new local pre-commit hook now run `python scripts/check_examples_readme.py`; pre-commit hook scoped to `examples/README.md` and `examples/*.vera`.
Docs & Metadata `CHANGELOG.md`, `HISTORY.md`, `README.md`, `TESTING.md`, `CONTRIBUTING.md`, `CLAUDE.md`, `ROADMAP.md`, `pyproject.toml`, `vera/__init__.py`	Docs and metadata bumped to v0.0.107 and updated to document/add the new validation step; pre-commit hook count incremented where applicable.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Dev as Developer
  participant Git as Git
  participant PreCommit as Pre-commit
  participant CI as CI
  participant Script as Validator (check_examples_readme.py)
  participant FS as Repo Filesystem

  Dev->>Git: commit / push
  Git->>PreCommit: run local hooks
  PreCommit->>Script: execute validator
  Script->>FS: read `examples/README.md`
  Script->>FS: read referenced `examples/*.vera`
  Script-->>PreCommit: exit 0 / exit 1
  Note over Git,CI: On push, CI workflow triggers
  Git->>CI: start `lint` job
  CI->>Script: run validation step
  Script->>FS: read README and .vera files
  Script-->>CI: exit 0 on pass, exit 1 on failure

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Add run instruction for examples/http.vera in README #362 — Adds a vera run instruction to examples/README.md that this validator will parse and verify.

Suggested labels

compiler, ci, docs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title precisely describes the main feature: adding validation for vera run commands in examples/README.md, referencing the linked issue `#361`.
Linked Issues check	✅ Passed	All requirements from issue `#361` are met: markdown parsing, vera run extraction, file existence verification, and --fn validation are implemented in scripts/check_examples_readme.py, wired into CI and pre-commit.
Out of Scope Changes check	✅ Passed	All changes are narrowly scoped to implementing issue `#361`: validation script, CI/pre-commit integration, documentation updates, and version bumps are all directly related to the stated objective.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/check-examples-readme-361

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

codecov · 2026-04-07T14:19:21Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.32%. Comparing base (3995d25) to head (4e19456).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #444   +/-   ##
=======================================
  Coverage   90.32%   90.32%           
=======================================
  Files          49       49           
  Lines       19335    19335           
  Branches      220      220           
=======================================
  Hits        17464    17464           
  Misses       1867     1867           
  Partials        4        4

Flag	Coverage Δ
javascript	`50.56% <ø> (ø)`
python	`95.27% <100.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Add hook table row, update lint job description, and remove #361 from the open CI issues table now that it is resolved. Co-Authored-By: Claude <noreply@anthropic.invalid>

aallan · 2026-04-07T14:21:00Z

@coderabbitai review

coderabbitai · 2026-04-07T14:21:06Z

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@scripts/check_examples_readme.py`:
- Around line 36-50: The parse_run_command function can IndexError if the
command contains no file path; update parse_run_command to defensively validate
the trimmed `rest` before calling `rest.split()[0]`: after stripping and
removing the "--" args, check if `rest` is empty or contains only
flags/whitespace, and if so raise a clear ValueError (or return a controlled
sentinel) that includes the original `cmd`; preserve existing `--fn` extraction
(fn_match) and only read the file_path when the check passes so you never call
rest.split()[0] on an empty list.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: b8351657-89c1-44c5-96eb-42225aae410d

📥 Commits

Reviewing files that changed from the base of the PR and between 3995d25 and 39ceda5.

📒 Files selected for processing (6)

.github/workflows/ci.yml
.pre-commit-config.yaml
CLAUDE.md
CONTRIBUTING.md
TESTING.md
scripts/check_examples_readme.py

If rest is empty after stripping (e.g. a malformed 'vera run ' cell), rest.split()[0] raises a cryptic IndexError. Check first and raise a ValueError with the original command string for clear diagnostics. Co-Authored-By: Claude <noreply@anthropic.invalid>

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@scripts/check_examples_readme.py`:
- Around line 78-87: parse_run_command can raise ValueError and currently will
crash the script; wrap the call to parse_run_command(cmd) inside a try/except
ValueError block in the loop over commands (the same loop that assigns lineno,
cmd and computes vera_file) and on exception append a failure to failures
(including lineno, the original cmd and the exception message) and continue, so
malformed commands are reported consistently instead of producing a traceback.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 0d6016fb-52a3-47b3-bc67-16d07f1588a7

📥 Commits

Reviewing files that changed from the base of the PR and between 39ceda5 and cab6320.

📒 Files selected for processing (1)

scripts/check_examples_readme.py

The previous commit added the raise but left the call site unguarded, so a malformed command would still crash with a traceback. Wrap the call in try/except and append to failures for consistent reporting. Co-Authored-By: Claude <noreply@anthropic.invalid>

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@scripts/check_examples_readme.py`:
- Around line 42-52: The parsing assumes the first token is the file path but
never removes the parsed --fn argument from rest, so commands like "vera run
--fn foo examples/bar.vera" yield tokens[0] == "--fn"; modify the logic after
detecting fn_match (variable fn_match and fn_name) to strip the entire matched
substring (the "--fn" plus its value) from rest before splitting into tokens,
then recompute tokens and set file_path from the first non-empty token; ensure
the same behavior when --fn is absent.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: f343b43b-35e6-4f26-be31-ee3a2fd1fe04

📥 Commits

Reviewing files that changed from the base of the PR and between cab6320 and 0f8b1e1.

📒 Files selected for processing (1)

scripts/check_examples_readme.py

- Remove #361 from ROADMAP CI tooling table (completed) - Add v0.0.107 one-liner to HISTORY.md Stage 9 table - Add v0.0.107 CHANGELOG entry with link reference - Bump version in vera/__init__.py, pyproject.toml, README.md, ROADMAP.md - Regenerate site assets Co-Authored-By: Claude <noreply@anthropic.invalid>

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@HISTORY.md`:
- Line 222: The HISTORY.md entry added for "v0.0.107" requires updating the
aggregate count under the "By the numbers" section (the "106 tagged releases"
total) to reflect the new release; find the "By the numbers" summary and change
the tagged releases total from 106 to 107 so the document remains consistent
with the newly added "v0.0.107" entry.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 136fb850-1aab-4205-995c-95d9ac5e18ec

📥 Commits

Reviewing files that changed from the base of the PR and between 0f8b1e1 and 52ebe08.

⛔ Files ignored due to path filters (4)

docs/index.md is excluded by !docs/**
docs/llms-full.txt is excluded by !docs/**
docs/llms.txt is excluded by !docs/**
docs/sitemap.xml is excluded by !docs/**

📒 Files selected for processing (6)

CHANGELOG.md
HISTORY.md
README.md
ROADMAP.md
pyproject.toml
vera/__init__.py

check_version_sync.py checks docs/index.html separately from the site assets regenerated by build_site.py — this one needs a manual update. Co-Authored-By: Claude <noreply@anthropic.invalid>

Co-Authored-By: Claude <noreply@anthropic.invalid>

docs: document check_examples_readme.py in TESTING.md

39ceda5

Add hook table row, update lint job description, and remove #361 from the open CI issues table now that it is resolved. Co-Authored-By: Claude <noreply@anthropic.invalid>

coderabbitai Bot reviewed Apr 7, 2026

View reviewed changes

Comment thread scripts/check_examples_readme.py

coderabbitai Bot reviewed Apr 7, 2026

View reviewed changes

Comment thread scripts/check_examples_readme.py

coderabbitai Bot reviewed Apr 7, 2026

View reviewed changes

Comment thread scripts/check_examples_readme.py

coderabbitai Bot reviewed Apr 7, 2026

View reviewed changes

Comment thread HISTORY.md

aallan and others added 2 commits April 7, 2026 16:08

fix: bump version to v0.0.107 in docs/index.html

68a4b70

check_version_sync.py checks docs/index.html separately from the site assets regenerated by build_site.py — this one needs a manual update. Co-Authored-By: Claude <noreply@anthropic.invalid>

fix: update tagged release count to 107 in HISTORY.md

4e19456

Co-Authored-By: Claude <noreply@anthropic.invalid>

aallan merged commit fae3c7b into main Apr 7, 2026
19 checks passed

aallan deleted the feat/check-examples-readme-361 branch April 7, 2026 15:38

Conversation

aallan commented Apr 7, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related PRs

Suggested labels

❌ Failed checks (1 warning)

Uh oh!

codecov Bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

aallan commented Apr 7, 2026

Uh oh!

coderabbitai Bot commented Apr 7, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

aallan commented Apr 7, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Apr 7, 2026 •

edited

Loading

codecov Bot commented Apr 7, 2026 •

edited

Loading