Address CR outside-diff findings on PR #70

aallan · claude · aallan · commit a5ba66a08b13 · 2026-05-22T12:46:04.000+01:00
Two of CR's three outside-diff findings on the latest review: 1. `_ailang_literal(value) -> str` was missing the parameter type hint on `value`. One-character fix matching the project's "type hints everywhere" rule from CLAUDE.md. The sibling `_aver_literal` has the same gap and predates this PR — that's a "do next time we touch the Aver path" mental note rather than scope-creep here. 2. Per-test subprocess failures in `_evaluate_aver_code` and `_evaluate_ailang_code` silently `continue` without capturing stderr — unlike the Python/TypeScript evaluators which record stderr into `ProblemResult.error_message`. Filed as #72 with a shared-helper refactor proposal that fixes Aver and AILANG consistently. Roadmap'd under Milestone 1; not blocking this PR. The third outside-diff finding (`AILANG_RESULTS.md:74` version pin inconsistency) becomes moot once the file is removed per ask #3 in the consolidated review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
diff --git a/ROADMAP.md b/ROADMAP.md
@@ -29,6 +29,7 @@
 - [x] Strengthen postconditions to catch slot-swap bugs (issue #14)
 - [ ] Improve SKILL.md coverage of where blocks (issue #15)
 - [x] Test coverage ([issue #5](https://github.com/aallan/vera-bench/issues/5), ongoing — target 90%) — CI enforces 80% floor via `--cov-fail-under=80` in [ci.yml](.github/workflows/ci.yml), current coverage shown by [![codecov](https://codecov.io/gh/aallan/vera-bench/graph/badge.svg)](https://codecov.io/gh/aallan/vera-bench)
+- [ ] Per-test subprocess-failure diagnostics — Aver and AILANG evaluators currently `continue` on per-test failures without capturing stderr, unlike the Python/TypeScript paths. Small shared-helper refactor (issue [#72](https://github.com/aallan/vera-bench/issues/72))
 
 ## Milestone 2: Longitudinal tracking
 
diff --git a/vera_bench/runner.py b/vera_bench/runner.py
@@ -593,7 +593,7 @@ def _strip_ailang_main(code: str) -> str:
     return "\n".join(out)
 
 
-def _ailang_literal(value) -> str:
+def _ailang_literal(value: object) -> str:
     """Convert a Python value to an AILANG literal expression."""
     if isinstance(value, bool):
         return "true" if value else "false"
