Fetch SKILL.md from veralang.dev instead of caching locally by aallan · Pull Request #22 · aallan/vera-bench

aallan · 2026-03-30T15:14:26Z

Summary

The cached context/SKILL.md drifted when vera updated to v0.0.105 (typed holes). Instead of maintaining a stale copy, fetch from the canonical source at runtime.

Behaviour

Default: fetches from https://veralang.dev/SKILL.md automatically
Override: --skill-md /path/to/local/SKILL.md for testing local changes
Non-Vera languages: SKILL.md not fetched (Python/TypeScript don't need it)

Changes

prompts.py: load_skill_md() now accepts URL or local path, defaults to veralang.dev
cli.py: removed hardcoded context/SKILL.md fallback, prints source being used
Deleted context/SKILL.md (1,773 lines of cached content that drifted)
README: documents auto-fetch and --skill-md override
CLAUDE.md: updated SKILL.md reference

Test plan

322 tests pass
Ruff + security lint clean
vera-bench run --model X --problem VB-T1-001 fetches from URL
vera-bench run --model X --skill-md ./local.md uses local file

Generated with Claude Code

Summary by CodeRabbit

New Features
- SKILL.md is now fetched automatically from the online language reference during benchmark runs
- Added a command-line option to override with a local SKILL.md via --skill-md
- Console now reports the SKILL.md source and a short content hash
Documentation
- Updated docs and README to describe online fetching behaviour and the local-override workflow

The cached context/SKILL.md drifted when vera updated to v0.0.105 (typed holes). Instead of maintaining a stale copy: - load_skill_md() now fetches from https://veralang.dev/SKILL.md by default, or reads a local file when --skill-md is provided - Deleted context/SKILL.md (the cached copy) - CLI prints which SKILL.md source is being used - --skill-md flag still works for local overrides (e.g. testing unreleased language features) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

codecov · 2026-03-30T15:15:49Z

Codecov Report

❌ Patch coverage is 21.05263% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.29%. Comparing base (5a99651) to head (8f15e3f).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
vera_bench/prompts.py	28.57%	10 Missing ⚠️
vera_bench/cli.py	0.00%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #22      +/-   ##
==========================================
- Coverage   66.95%   66.29%   -0.66%     
==========================================
  Files          10       10              
  Lines        1053     1068      +15     
==========================================
+ Hits          705      708       +3     
- Misses        348      360      +12

Flag	Coverage Δ
python	`66.29% <21.05%> (-0.66%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

coderabbitai · 2026-03-30T15:17:14Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 653cce8a-1c9e-41b0-adbb-f5d850bd1152

📥 Commits

Reviewing files that changed from the base of the PR and between a4aac20 and 8f15e3f.

📒 Files selected for processing (1)

vera_bench/prompts.py

📝 Walkthrough

Walkthrough

The default SKILL.md source was changed from a local snapshot to fetching https://veralang.dev/SKILL.md at runtime; a --skill-md /path/to/SKILL.md override was added. CLI now reports the SKILL.md source and a truncated content hash; prompts.load_skill_md accepts HTTP or local sources with error handling.

Changes

Cohort / File(s)	Summary
Documentation `CLAUDE.md`, `README.md`	Updated docs to state SKILL.md is fetched from `https://veralang.dev/SKILL.md` by default and document the `--skill-md /path/to/SKILL.md` override and CLI example.
Prompts / SKILL loader `vera_bench/prompts.py`	Added `SKILL_MD_URL` constant and changed `load_skill_md` signature to `source: str
CLI integration `vera_bench/cli.py`	When `language == "vera"`, derive `source` from `skill_md` or `SKILL_MD_URL`, load content via new `load_skill_md`, compute a truncated SHA‑256 of content, and print `SKILL.md: {source} ({content_hash})`. Previously used a local `context/SKILL.md` path when unset.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Add --language python for cross-language LLM comparison #11: Modifies SKILL.md handling and CLI behaviour in the same modules (vera_bench/prompts.py, vera_bench/cli.py), likely overlapping changes.
Implement LLM runner harness (Phase 2) #3: Earlier changes to CLI run flow and load_skill_md signature/behaviour touching the same files.

Suggested labels

harness, docs

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarises the main change: switching from cached local SKILL.md to runtime fetching from veralang.dev, which is the core objective addressed across all modified files.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feature/fetch-skill-md

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@vera_bench/cli.py`:
- Around line 131-138: When language == "vera" and SKILL.md is selected/fetched,
record the SKILL.md source and/or a content hash in the run results metadata so
benchmarks are reproducible; locate the code around the language check
(variables: language, skill_md, SKILL_MD_URL, skill_content) and after calling
load_skill_md(skill_md) add the chosen source string (source = str(skill_md) if
skill_md else SKILL_MD_URL) and a content checksum (compute hash of
skill_content) into the JSONL results metadata writer so each result entry
includes SKILL.md_source and/or SKILL.md_hash fields; keep console.print for
human visibility but ensure the metadata fields are written programmatically
where results are assembled/saved.

In `@vera_bench/prompts.py`:
- Around line 30-33: The network fetch using urllib.request.urlopen on variable
source_str must include a timeout and robust error handling: update the code in
prompts.py where source_str.startswith("http") and the urllib.request.urlopen
call to pass a reasonable timeout (e.g., 5–10 seconds) and wrap the open/read in
a try/except that catches urllib.error.URLError and urllib.error.HTTPError (and
optionally socket.timeout), then log or raise a clear, user-friendly error
message including the URL and the original exception; ensure the
resp.read().decode(...) only runs on successful responses and that resources are
properly closed (keep the with-block pattern).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1841b5a9-8f7b-4113-b978-e144dbb17122

📥 Commits

Reviewing files that changed from the base of the PR and between 5a99651 and 31e393f.

⛔ Files ignored due to path filters (1)

context/SKILL.md is excluded by !context/**

📒 Files selected for processing (4)

CLAUDE.md
README.md
vera_bench/cli.py
vera_bench/prompts.py

- urlopen now has 10s timeout and catches URLError/OSError with actionable error message suggesting --skill-md - CLI prints SHA-256 hash (first 12 chars) of SKILL.md content for reproducibility tracking Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@vera_bench/prompts.py`:
- Line 5: Add an explicit import for urllib.error so the exception reference to
urllib.error.URLError resolves at runtime: update the imports that currently
include only urllib.request to also import urllib.error (so the except clause
referencing urllib.error.URLError can catch real network errors instead of
causing AttributeError); locate the import line where urllib.request is imported
and add the urllib.error import alongside it.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 918cdfc6-fab3-4b73-b149-bfa556f9acbf

📥 Commits

Reviewing files that changed from the base of the PR and between 31e393f and a4aac20.

📒 Files selected for processing (2)

vera_bench/cli.py
vera_bench/prompts.py

Relying on urllib.request to transitively import urllib.error is a CPython implementation detail, not a language guarantee. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

coderabbitai Bot reviewed Mar 30, 2026

View reviewed changes

Comment thread vera_bench/cli.py

Comment thread vera_bench/prompts.py Outdated

coderabbitai Bot reviewed Mar 30, 2026

View reviewed changes

Comment thread vera_bench/prompts.py

Add explicit import urllib.error

8f15e3f

Relying on urllib.request to transitively import urllib.error is a CPython implementation detail, not a language guarantee. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

aallan merged commit 2396d48 into main Mar 30, 2026
10 checks passed

aallan deleted the feature/fetch-skill-md branch March 30, 2026 15:48

coderabbitai Bot mentioned this pull request Apr 16, 2026

Fix Cloudflare 403 on Aver llms.txt fetch #58

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fetch SKILL.md from veralang.dev instead of caching locally#22

Fetch SKILL.md from veralang.dev instead of caching locally#22
aallan merged 3 commits into
mainfrom
feature/fetch-skill-md

aallan commented Mar 30, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

codecov Bot commented Mar 30, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Mar 30, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

aallan commented Mar 30, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behaviour

Changes

Test plan

Summary by CodeRabbit

Uh oh!

codecov Bot commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

coderabbitai Bot commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

aallan commented Mar 30, 2026 •

edited by coderabbitai Bot

Loading

codecov Bot commented Mar 30, 2026 •

edited

Loading

coderabbitai Bot commented Mar 30, 2026 •

edited

Loading