Document Aver support and add comparison language guide#52
Merged
Conversation
3 tasks
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds Aver as a comparison language: introduces a language‑agnostic Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #52 +/- ##
=======================================
Coverage 83.06% 83.06%
=======================================
Files 10 10
Lines 1346 1346
=======================================
Hits 1118 1118
Misses 228 228
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@CLAUDE.md`:
- Line 13: The document references an invalid external URL for "Aver's llms.txt"
(the line containing "**Aver's llms.txt** is fetched from
`https://averlang.org/llms.txt` at runtime"); update this to a valid Aver
resource or remove the reference: replace the broken
`https://averlang.org/llms.txt` link with a correct repo or docs URL such as
`https://github.com/jasisz/aver` or `https://jasisz.github.io/aver-language/`,
and if `llms.txt` is not provided by Aver, remove or reword the sentence to
point to the appropriate file/resource in the Aver project.
In `@CONTRIBUTING.md`:
- Around line 27-47: The .coderabbit.yaml path_filters are missing the Aver
exclusions referenced by CONTRIBUTING.md step 8; update the path_filters block
in .coderabbit.yaml to include the patterns !**/*.av and !solutions/aver/**
alongside the existing !**/*.vera, !solutions/python/** and
!solutions/typescript/** entries so the documented guidance for excluding Aver
solutions from CodeRabbit is enforced.
In `@scripts/run_full_benchmark.py`:
- Around line 237-245: The "Aver baselines" benchmark entry currently uses
"--language", "aver" which will raise NotImplementedError; either remove or
disable that tuple from the benchmarks list in run_full_benchmark.py (the tuple
starting with "Aver baselines") or replace "aver" with a supported language
("python" or "typescript"); if you prefer to keep the target, implement support
for "aver" in the baseline runner by updating vera_bench/baseline_runner.py to
accept "aver" alongside "python"/"typescript" and provide the corresponding
baseline implementation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 3cac3557-4568-4981-9cec-983c0d4c7da0
📒 Files selected for processing (7)
CHANGELOG.mdCLAUDE.mdCONTRIBUTING.mdDESIGN.mdREADME.mdROADMAP.mdscripts/run_full_benchmark.py
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@CLAUDE.md`:
- Line 91: Replace the incorrect Aver repo URL in the CLAUDE.md sentence that
mentions Aver and its reference doc (the text containing
"https://github.com/pchj/aver-lang" and the surrounding mention of `llms.txt`
and the `aver` command) with the official repository URL
"https://github.com/jasisz/aver" so the link points to the correct Aver project.
- Line 13: Replace the incorrect GitHub URL string
"https://github.com/pchj/aver-lang" in CLAUDE.md with the correct repository
"https://github.com/jasisz/aver" (this is the link referenced alongside the Aver
resource mention such as "Aver's llms.txt"); locate the existing URL text in the
file and update the link so both occurrences (line ~13/91 mentions) point to the
correct Aver repo.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: d31bd782-5f13-452d-8f83-65b6505e5ff6
📒 Files selected for processing (1)
CLAUDE.md
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
README.md (1)
136-150:⚠️ Potential issue | 🟡 MinorUpdate the full-suite target count to match current behaviour.
The comment on Line 149 still says “all 6 targets”, but this PR’s benchmark scope is now 8 targets. Please update that line to avoid misleading run expectations.
Suggested doc fix
-# Or run the full benchmark suite (all 6 targets) with one command +# Or run the full benchmark suite (all 8 targets) with one command python scripts/run_full_benchmark.pyAs per coding guidelines,
**/*.md: “Review for factual accuracy and broken links. Do not review for grammar or prose style.”🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@README.md` around lines 136 - 150, The README's final comment "all 6 targets" is outdated; update the text near the "python scripts/run_full_benchmark.py" example to reflect the current benchmark scope (change "all 6 targets" to "all 8 targets") so the run_full_benchmark.py invocation accurately describes behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@README.md`:
- Around line 136-150: The README's final comment "all 6 targets" is outdated;
update the text near the "python scripts/run_full_benchmark.py" example to
reflect the current benchmark scope (change "all 6 targets" to "all 8 targets")
so the run_full_benchmark.py invocation accurately describes behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 428ca627-503b-4a2d-8212-086529dd07ed
📒 Files selected for processing (2)
CLAUDE.mdREADME.md
Owner
Author
|
Fixed in f95abdc — README now says "all 8 targets" to match |
|
Tip For best results, initiate chat on the files or code changes.
🧠 Learnings used |
Update README, CLAUDE.md, DESIGN.md, CONTRIBUTING.md, and ROADMAP.md to document Aver as a comparison language, the description_neutral field, and the pattern for adding new languages. Add Aver targets to the full benchmark runner script. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The pchj/aver-lang URL is a 404. The correct repo is github.com/jasisz/aver (the PR #48 contributor repo). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
f95abdc to
ebd1dc4
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@CONTRIBUTING.md`:
- Line 20: The CONTRIBUTING.md requires description_neutral but the validator
omits it; update vera_bench/validate.py by adding "description_neutral" to the
REQUIRED_FIELDS set/tuple used by the problem JSON validation (where
REQUIRED_FIELDS is declared and referenced, e.g., in validate_problem or
validate_json logic) so that vera-bench validate enforces this field; run
existing unit tests or add a small test to ensure missing description_neutral
causes validation to fail.
In `@scripts/run_full_benchmark.py`:
- Around line 16-24: Update the module's top docstring sentence that currently
says "all 6 targets" to reflect the new count "all 8 targets" so it matches the
listed targets; locate the top-level module docstring in
scripts/run_full_benchmark.py (the opening sentence of the file) and change the
numeric text to "8" or "eight" to keep the docstring and the enumerated list
consistent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 104c0be5-80f8-4345-9c6d-a2e479177ee7
📒 Files selected for processing (7)
CHANGELOG.mdCLAUDE.mdCONTRIBUTING.mdDESIGN.mdREADME.mdROADMAP.mdscripts/run_full_benchmark.py
The validator now enforces description_neutral on problem JSONs, matching what CONTRIBUTING.md documents. Also fix the stale '6 targets' in run_full_benchmark.py opening docstring. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
description_neutralfield documentation and rationalescripts/run_full_benchmark.py(6 → 8 targets)Depends on #48 — CHANGELOG.md will need a rebase after #48 merges (both add to
[Unreleased]). All other files are non-overlapping.Test plan
ruff check .andruff format --check .pass🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation