Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.coderabbit.yaml:
- Line 145: Remove the trailing blank line at EOF so the file ends immediately
after the simplify: block's enabled: false line; open .coderabbit.yaml, locate
the simplify: section (the enabled: false setting) and delete the extra empty
newline after that line so the file no longer has a blank line at the end.

In `@problems/tier1/VB_T1_005_min_of_two.json`:
- Line 22: Update the "notes" field in the JSON for the min_of_two problem to
explicitly state the De Bruijn slot mapping (e.g., "@Int.0 = rightmost
parameter, `@Int.1` = leftmost parameter") so readers/implementers know the exact
ordering; modify the existing notes entry (the "notes" property) to include this
precise mapping and a short reminder that `@Int.N` refers to De Bruijn indices for
parameters.

In `@problems/tier1/VB_T1_009_max_of_three.json`:
- Line 9: The postcondition in VB_T1_009_max_of_three.json currently only
asserts lower bounds ("@Int.result >= `@Int.2`", "@Int.result >= `@Int.1`",
"@Int.result >= `@Int.0`") and must be strengthened so the result is the maximum
of the three inputs; change the ensures to also require that `@Int.result` equals
one of the inputs (i.e. add a clause requiring `@Int.result` == `@Int.0` OR
`@Int.result` == `@Int.1` OR `@Int.result` == `@Int.2` or the equivalent JSON
expression) so the contract prevents arbitrarily larger values and exactly
matches the intended semantics of max_of_three.

In `@problems/tier2/VB_T2_003_greeting.json`:
- Line 23: The "test_cases" array in VB_T2_003_greeting.json was intentionally
emptied; update the file to document why by adding a concise explanation in the
"notes" field describing that the string argument/output test was removed due to
known limitations in runtime handling for `vera run --fn` (e.g., inconsistent
string quoting/escaping) and state that validation is intentionally skipped for
this function until the CLI/runtime string handling is fixed; reference the JSON
keys "test_cases" and "notes" and include the CLI command `vera run --fn` in the
note so future maintainers can trace the rationale.

In `@problems/tier2/VB_T2_004_is_empty_string.json`:
- Line 14: The test manifest's "test_cases" array is empty; add runtime test
cases to validate behavior (e.g., one case with args [""], expected true, and
one with args ["hello"], expected false) by populating the "test_cases" field in
VB_T2_004_is_empty_string.json, or if tests are intentionally omitted, add an
explanatory entry in the "notes" field explaining the limitation; ensure the
JSON keys "test_cases" and "notes" are updated accordingly.

In `@problems/tier2/VB_T2_005_contains_substring.json`:
- Line 14: The "test_cases" array is empty so automated validation is skipped;
populate the "test_cases" key with representative JSON test objects for the
string-contains function (each object should include "args": [string, substring]
and "expected": boolean), e.g. add cases that cover positive match, negative
match, empty substring, and case-sensitivity (e.g. {"args":["hello
world","world"],"expected":true}, {"args":["hello","x"],"expected":false},
{"args":["abc",""],"expected":true},
{"args":["Hello","hello"],"expected":false}).
- Around line 7-11: The ensures clause for contains_substring is too weak
(["true"]); update the "ensures" in the contract to express the function's
behavior by relating its result to the string_contains builtin: ensure the
postcondition states that the function returns the same boolean as
string_contains called with the same arguments (preserve the argument order used
in the implementation), e.g., an ensures that equates the function's return
value to string_contains(arg1, arg2) so Z3 can verify the wrapper semantics for
contains_substring.

In `@problems/tier2/VB_T2_006_join_strings.json`:
- Line 14: The problem JSON is missing concrete runtime examples in the
"test_cases" array for the join-strings task (VB_T2_006), so add representative
cases covering: an empty list (expect ""), a single-element list (expect that
element), a multi-element list with a typical separator (e.g., ["a","b","c"] +
"," → "a,b,c"), and a case with an empty separator (e.g., ["a","b"] + "" →
"ab"); ensure each test_case object matches the schema used elsewhere (inputs
matching the contract keys and the exact expected output strings) so the
examples align with the canonical solution and enforce correct joining
semantics.

In `@problems/tier2/VB_T2_007_double_elements.json`:
- Line 14: The test_cases array is empty while the contract in the JSON is a
tautology ("true"), so add concrete executable test cases that cover array
mapping behaviour to prevent false positives; update the "test_cases" entry in
VB_T2_007_double_elements.json with several input/output pairs that reflect the
canonical solution's expected results (e.g., various arrays including empty,
single-element, duplicated values and typical even/odd cases) so the JSON's test
suite validates the mapping logic rather than relying on the tautological
contract.

In `@problems/tier2/VB_T2_008_count_positives.json`:
- Around line 9-17: The contract currently only asserts non-negativity and has
no runtime tests: add concrete test_cases for the entry_point "count_positives"
to validate counting semantics (e.g., empty array -> 0, all positives -> length,
all negatives -> 0, mixed values -> correct count, zeros treated as non-positive
if intended). Update the JSON "test_cases" array with objects that pass an input
array and the expected integer result, and optionally strengthen the "ensures"
to a more specific property if desired, targeting the "count_positives" function
name so the benchmark verifies behavior rather than just type-level
non-negativity.

In `@problems/tier2/VB_T2_009_to_upper.json`:
- Line 14: Add concrete test cases to the "test_cases" array in
VB_T2_009_to_upper.json to validate uppercase behavior: include (1) a mixed-case
ASCII input that asserts the correctly uppercased string, (2) an
already-uppercase input that asserts the same string is unchanged, and (3) an
empty string that asserts empty output; ensure each test case's expected output
matches the canonical solution and that any contract entries in the JSON reflect
these functional expectations (e.g., function name/contract for to_upper or
equivalent).

In `@problems/tier2/VB_T2_010_sum_positives.json`:
- Around line 9-15: The spec for the entry_point sum_positives has no behavioral
tests (ensures is "true" and test_cases is empty); add representative test_cases
verifying correct outputs for (1) mixed-sign array (e.g., positives and
negatives) with expected sum of positives, (2) all non-positive array (expected
0), and (3) empty array (expected 0), making sure each test case supplies the
input array under the same parameter name the solution expects and the correct
expected output; keep the entry_point "sum_positives" and ensure test case
objects match the repository's test schema so the checker exercises the
function.

In `@problems/tier3/VB_T3_005_tree_sum.json`:
- Line 14: The problem JSON currently has an empty "test_cases" array and a
permissive "ensures": ["true"]; add canonical ADT recursion test vectors and a
stricter ensures clause: populate "test_cases" with at least three cases (single
leaf, balanced branch, nested branch) each including an input tree and the
expected numeric sum, and update "ensures" to assert the function's return
equals the expected sum (instead of "true") so the contract and canonical
solution align; target the "test_cases" field and the "ensures" array when
making these edits.

In `@problems/tier3/VB_T3_006_option_unwrap_or.json`:
- Line 14: The "test_cases" array is empty; either document that ADT-style
argument passing (e.g., passing Some(42) to vera via `vera run --fn`) is
unsupported or add concrete runtime tests to validate behavior. Update the
JSON's "test_cases" field: if documenting the limitation, add a descriptive note
in the problem metadata explaining that `vera run --fn` cannot accept
constructed ADT values like Some(42); otherwise add one or more test cases that
invoke the target function with ADT-like inputs (e.g., an input representing
Some(42) and the expected output) so runtime behavior is verified.

In `@problems/tier3/VB_T3_008_tree_count_leaves.json`:
- Around line 9-14: The spec currently allows a constant returning 1 to pass
because it only has "ensures": ["@Nat.result >= 1"] and no test_cases; update
the JSON for entry_point "tree_count_leaves" to include concrete test_cases that
exercise the ADT and required semantics (reference the entry_point name and the
ensures clause). Add multiple fixtures: a single-leaf tree (expect 1), a tree
with two leaves (expect 2), a deeper tree with multiple internal nodes (expect
the correct leaf count), and a balanced tree (expect its leaf count), ensuring
each test case's input uses the same tree ADT used by the canonical solution and
that the expected outputs are exact integers matching the canonical
implementation. Ensure test_cases array is non-empty and covers edge and typical
trees so a hard-coded constant cannot satisfy all cases.

In `@problems/tier4/VB_T4_006_list_reverse.json`:
- Line 17: Update the note to distinguish ADT field indices from
function-parameter indices: clarify that in the helper reverse_acc(acc, xs) the
indices refer to parameters so `@List.0` = xs and `@List.1` = acc, and separately
state that in a Cons match the ADT fields are `@Int.0` = head and `@List.0` = tail;
mention reverse_acc, acc, xs and Cons so readers can locate the relevant
semantics.

In `@problems/tier4/VB_T4_009_list_nth.json`:
- Line 14: The JSON for problem VB_T4_009_list_nth has an empty "test_cases"
because the test runner cannot pass constructed List ADT values; update the
problem metadata by leaving "test_cases" empty but adding a short explanatory
note (e.g., a "note" or "remarks" field) that states the vera run --fn
limitation for ADT-based problems and mirrors the wording used in other ADT
problems so users understand why no runnable test cases are present.

In `@problems/tier5/VB_T5_004_accumulator.json`:
- Around line 7-11: The contract's precondition is redundant because "@Nat.0 >=
0" is guaranteed by the `@Nat` type; update the "contracts" object to remove this
tautological requirement by replacing the "requires" array with ["true"] (keep
the existing "ensures" and "effects" entries unchanged) so the contract reads
requires: ["true"].

In `@problems/tier5/VB_T5_005_checked_index.json`:
- Around line 7-11: The current postcondition in the "contracts" JSON uses the
trivial ensures ["true"]; update the "ensures" field to a meaningful predicate
that constrains valid results (e.g., ensure the returned index or value lies
within expected bounds and that no exception is thrown when the input index is
within bounds). Edit the "contracts" -> "ensures" array to replace ["true"] with
a precise expression such as a bound check (for example "result >= 0 && result <
length(array)" or "index_in_range ==> no_throw" depending on the function's
result semantics) so verification can prove correct behavior when the index is
valid.

In `@problems/tier5/VB_T5_009_state_max.json`:
- Around line 8-10: The postcondition "ensures": ["true"] is too weak; replace
it with a concrete relation tying the returned value to input n (e.g., assert
that the function return equals the maximum over the first n elements and/or
that it is ≥ each element for all indices < n and equals some array element).
Update the "ensures" clause in VB_T5_009_state_max.json to express that relation
(use quantifiers or an explicit max specification referencing input n and the
return value, e.g., ret == max(a[0..n-1]) or (forall i < n: ret >= a[i] &&
exists j < n: ret == a[j])).

In `@problems/tier5/VB_T5_010_safe_head.json`:
- Around line 9-14: The spec for the entry_point safe_head is untestable because
"ensures" is just "true" and "test_cases" is empty; update the JSON so the
postcondition reflects the intended behavior (e.g. return equals the first
element for non-empty arrays or -1 when empty) and add at least two executable
test_cases covering an empty array and a non-empty array (with expected outputs
matching the canonical safe_head behavior) so incorrect implementations cannot
pass; modify the "ensures" clause to reference the function result (e.g., result
== arr[0] || (arr == [] && result == -1)) and include concrete test_cases
exercising both branches.

In `@README.md`:
- Around line 73-81: Update the repository so the declared license is
consistent: decide whether the project uses MIT or Apache-2.0, then change the
license text in README (the license paragraph and the SPDX link) and the license
metadata in pyproject.toml (the `license = "..."` value) to match that chosen
license; also correct the project name in README by replacing the word "Vera"
with "VeraBench" where the license sentence appears so it reads "VeraBench is
licensed". Ensure both files reference the exact same license identifier (e.g.,
MIT or Apache-2.0) and update the LICENSE file if needed to match.

In `@vera_bench/__init__.py`:
- Line 3: The module-level __version__ string duplicates the version in
pyproject.toml; either remove the fixed __version__ variable or replace it with
a single-source implementation that reads package metadata (e.g., set
__version__ = importlib.metadata.version("vera-bench") with a safe fallback for
older Pythons or when metadata is missing) so the CLI that uses
`@click.version_option`(package_name="vera-bench") and any other code use the same
authoritative version.

In `@vera_bench/baseline_runner.py`:
- Line 1: The module vera_bench/baseline_runner.py is a placeholder with only a
docstring; implement a baseline runner that provides functions to run Python and
TypeScript solutions with subprocess control, timeouts, and error handling: add
run_baseline(entry_path, timeout_seconds) as the main entry, implement
run_python_solution(script_path, timeout_seconds) to execute via sys.executable
and capture stdout/stderr and exit code, and implement
run_typescript_solution(project_dir, timeout_seconds) to run npm/yarn build (if
needed) and node on the compiled JS or use ts-node if available; for both
runners ensure subprocess.run is used with timeout, capture_output=True, check
for CalledProcessError/TimeoutExpired, and return a structured result
(exit_code, stdout, stderr, timed_out) while documenting functions and raising
no uncaught exceptions. Ensure function names run_baseline, run_python_solution,
and run_typescript_solution are present so callers can locate them.

In `@vera_bench/cli.py`:
- Line 38: The console.print call uses an unnecessary f-string literal with no
placeholders; in the call to console.print (the line containing
console.print(f"[yellow]vera-bench run is not yet implemented.[/yellow]"))
remove the leading "f" so the argument is a plain string literal
"[yellow]vera-bench run is not yet implemented.[/yellow]" to satisfy static
analysis.
- Line 48: The console.print call uses an unnecessary f-string prefix for a
literal with no placeholders; update the console.print invocation in
vera_bench/cli.py (the console.print(...) statement) by removing the leading "f"
so the argument is a plain string literal: console.print("[yellow]vera-bench
report is not yet implemented.[/yellow]").
- Around line 36-42: The run function currently prints only model, tier and
mode; update the stub for run(model, tier, problem, mode, skill_md, output_dir)
to also print the problem, skill_md, and output_dir parameters (showing a
sensible fallback like 'None' or 'not provided') so callers see all inputs for
debugging; modify the console.print block in run to include these fields and
keep formatting consistent with the existing model/tier/mode lines.

In `@vera_bench/metrics.py`:
- Line 1: The module vera_bench.metrics.py is currently a placeholder; implement
the core metric functions and exports: add compute_pass_rate(results),
compute_verification_rate(results), aggregate_performance(results,
by="suite"|"model"), and a helper normalize_metrics(results) with clear
docstrings, input/result type expectations, and error handling for empty/invalid
inputs; ensure these functions accept the benchmark result structures used
elsewhere in the repo, return numeric summaries or dicts, and export them (e.g.,
via __all__) so callers can import them and add unit tests exercising edge cases
(empty lists, missing fields, mixed statuses).

In `@vera_bench/prompts.py`:
- Around line 63-75: The build_fix_prompt function currently ignores its
original_code parameter so the LLM only receives error_output; update user_msg
in build_fix_prompt to include the original_code (e.g., insert a clear code
fence or a "vera\n{original_code}\n```" block before the error text) so the
prompt provides both the original code and the error; ensure you still return
the same dict with "system": SYSTEM_PROMPT and "user": user_msg.
- Line 5: Remove the unused import by deleting the top-level "import json" in
this module (the unused symbol "json") and run a quick search in the file
(prompts.py) to confirm nothing references json before committing; if you
intended to use JSON functionality, replace the unused import with the actual
usage or add a comment explaining why it remains.
- Around line 16-24: The _format_contracts function builds a list with explicit
for-loops; refactor it to use list comprehensions to construct the lines more
concisely: create lists for requires and ensures with comprehensions producing "
requires(...)" and "  ensures(...)" respectively, then append the effects line
(using contracts.get("effects", "pure")) and return the joined string. Keep the
same function name _format_contracts and preserve behavior for missing keys and
ordering (requires then ensures then effects).

In `@vera_bench/validate.py`:
- Around line 68-73: The JSON open/load block in the try/except uses
platform-default encoding; change the file open call in validate.py (the with
open(problem_path) used before json.load) to explicitly specify encoding="utf-8"
so the json.load reads UTF-8 consistently; keep the existing exception handling
that appends to result["errors"] and returns result on failure.
- Around line 22-27: The function find_vera_file currently returns None silently
when multiple .vera files match; update it to surface diagnostics by detecting
len(matches) > 1 and logging or raising an error that includes the conflicting
file paths (use the matches list to build a message). Specifically, in
find_vera_file add handling for multiple matches that logs (via an existing
logger) or raises a ValueError with a descriptive message listing the matched
Path objects (or their str() values) so callers can see which files conflicted;
keep the existing single-match and no-match behaviors otherwise.

In `@vera_bench/vera_runner.py`:
- Around line 114-126: The subprocess.run call inside run_fn currently omits the
explicit check argument; update the call in function run_fn (the subprocess.run
invocation in vera_runner.py) to include check=False so its behavior matches the
other two subprocess calls and then return the same RunResult(exit_code=...,
stdout=..., stderr=...).
- Around line 86-112: The subprocess.run call inside the verify method should
pass check=False to avoid raising CalledProcessError on non-zero exits; update
the subprocess.run invocation that uses cmd, timeout=self.timeout_verify,
capture_output=True, text=True to include check=False so the function can
inspect returncode and parse combined stdout/stderr without exceptions (refer to
the verify method, the cmd variable, and timeout_verify).
- Around line 62-84: In the check method, the subprocess.run call intentionally
inspects the return code instead of raising; update the call to
subprocess.run(...) inside vera_bench.vera_runner.VeraRunner.check (the
invocation that builds cmd = [self.vera, "check", "--json", str(file_path)]) to
include check=False so the intent is explicit and static analysis warnings are
silenced; keep capture_output, text, and timeout unchanged.

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/PULL_REQUEST_TEMPLATE.md:
- Around line 14-15: The current success boolean "ok" uses r["fields_ok"],
r["vera_found"], r["check_pass"] and r["errors"] but omits r["verify_pass"],
causing the checklist mismatch; update the success expression that sets ok (in
validate.py) to also require r["verify_pass"] (e.g., include r["verify_pass"] in
the conjunction) so vera verify failures fail validation, and adjust any
reported summary/exit code logic accordingly; alternatively, if you prefer
verify to be informational, update the PR template checklist to remove or
rephrase the "vera verify" requirement so it matches the existing validate.py
behavior.

In @.github/workflows/ci.yml:
- Around line 10-35: The validate job currently has no timeout and can hang
indefinitely; add a job-level timeout by adding a timeout-minutes field to the
validate job (e.g., timeout-minutes: 30) so the entire job will be cancelled if
it exceeds that duration; update the job block named "validate" that contains
steps running "python scripts/validate_problems.py" and "python
scripts/validate_solutions.py" (and any subprocess-invoking vera commands) to
include the timeout-minutes key with an appropriate value.
- Around line 36-61: Add a job-level timeout to the "test" GitHub Actions job to
prevent runaway pytest runs; in the "test" job definition (the job named test in
.github/workflows/ci.yml) add a timeout-minutes property (e.g., timeout-minutes:
30) at the same indentation level as strategy/runs-on so the entire job will be
cancelled after the specified time.
- Around line 33-34: The CI step named "Validate all canonical solutions execute
correctly" calls a non-existent script scripts/validate_solutions.py; either
remove this step or point it to the existing script
(scripts/validate_problems.py) or add the missing scripts/validate_solutions.py
implementation; update the job step that runs python
scripts/validate_solutions.py accordingly so the workflow uses an existing
script or the new implementation.

In @.github/workflows/dependabot.yaml:
- Around line 1-25: The Dependabot config file named "dependabot.yaml" with the
top-level key "updates" is in the wrong place; move the file out of the
workflows folder into the repository's .github directory (e.g., rename/move to
.github/dependabot.yaml) so Dependabot can discover it, and confirm the file
retains the existing keys (package-ecosystem, directory, schedule,
commit-message, labels, open-pull-requests-limit).

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/ci.yml:
- Around line 24-25: The workflow currently installs Vera from the repository
default branch; update the "Install vera compiler" steps in both the validate
and test jobs to pin to an immutable reference by changing the pip install URL
to include a release tag or commit SHA (e.g.,
git+https://github.com/aallan/vera.git@<tag-or-sha>) so the Vera CLI
output/schema remains stable across CI runs; ensure both occurrences of the
"Install vera compiler" step are updated to the same pinned tag or SHA.

In `@vera_bench/validate.py`:
- Around line 195-198: The overall success calculation omits the verification
result: update the boolean expressions that compute ok and passed (where ok =
r["fields_ok"] and r["vera_found"] and r["check_pass"] and not r["errors"]) to
also require r["verify_pass"]; likewise update the analogous passed computation
later to include r["verify_pass"] so a failed verification prevents all_ok from
remaining True and the process returning exit code 0.
- Around line 143-151: The current test counting logic uses runner.run_fn and
normalize_output but ignores run.exit_code; update the block that handles the
run result from runner.run_fn(vera_file, entry_point, args if args else None) to
first check run.exit_code == 0 before considering stdout: only when exit_code is
0 and normalized actual == expected_str should you increment
result["tests_pass"]; otherwise append an error to result["errors"] that
includes both the mismatch and the non‑zero exit code (e.g., "run(args): exit
code X, expected Y, got Z" or a similar clear message), leaving normalize_output
and result structure unchanged.
- Around line 158-167: The function run_validation currently computes repo_root
= Path(__file__).parent.parent and falls back to problems_dir/solutions_dir
under that path, which breaks when the package is installed from a wheel
(site-packages) and those directories are not present; also runner.run_fn() exit
codes are ignored when deciding pass/fail. Fix by removing the implicit
repo_root fallback: require callers/CLI to provide explicit problems_dir and
solutions_dir or load them via package resources (importlib.resources) if you
intend to ship the corpus with the package; validate existence of the resolved
problems_dir and solutions_dir early in run_validation and raise a clear error
if missing. Additionally, after calling runner.run_fn(...) check its exit code
(return value) and treat any non-zero exit as a test failure before considering
stdout comparison so failing executions are not marked as passing. Ensure
references to repo_root, problems_dir, solutions_dir, run_validation, and
runner.run_fn() are updated accordingly.

In `@vera_bench/vera_runner.py`:
- Around line 69-72: The code currently concatenates result.stdout +
result.stderr and calls json.loads on that (see combined and
json.loads(combined)), which breaks when --json writes valid JSON to stdout and
diagnostics to stderr; change the logic to try json.loads(result.stdout) first
and only if that fails, try json.loads(result.stderr) (do not concatenate the
two), and apply the same change in the verify() method so JSON payloads from
stdout are parsed preferentially while stderr is used as a fallback for
structured output.
- Around line 66-68: Add the required Ruff S603 suppression comments to each
subprocess.run invocation by appending "  # noqa: S603 - audited Vera CLI
invocation via argv list; no shell" to the subprocess.run call in the check(),
verify(), and run_fn() functions (the calls at the sites currently invoking
subprocess.run with list argv, timeout and exit handling). Ensure the exact
comment is added inline on the same line as the subprocess.run( call for each of
those three locations so the linter recognizes the audit suppression.

---

Duplicate comments:
In `@vera_bench/prompts.py`:
- Around line 69-79: The build_fix_prompt function currently ignores
original_code; update the user_msg in build_fix_prompt to include the previous
Vera source (use original_code) alongside the error_output (e.g., prepend
"Original code:\n{original_code}\n\n"), and ensure the function returns the
expected dict with 'system' and 'user' keys (populate 'user' with the new
user_msg). Modify the user_msg variable and the function return so callers of
build_fix_prompt receive both system and user entries that include the original
code to be repaired.

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/dependabot.yaml:
- Line 25: Remove the trailing blank line at the end of the YAML file so the
file ends immediately after the open-pull-requests-limit: 5 entry; open
.github/dependabot.yaml, delete the empty line following the final mapping (the
line after "open-pull-requests-limit: 5") and save to satisfy YAML linting.

In @.github/workflows/ci.yml:
- Around line 78-138: Add a consistent timeout for the remaining GitHub Actions
jobs by adding a timeout-minutes field to the lint, security, and
dependency-audit job definitions (e.g., timeout-minutes: 30) so long-running
steps like the security job’s checkout with fetch-depth: 0 and the
gitleaks/gitleaks-action@v2 step cannot hang indefinitely; update the job blocks
named lint, security, and dependency-audit to include the chosen timeout value.
- Around line 62-67: The "Run tests" step currently runs for all matrix entries
including Python 3.12, causing tests to run twice because "Run tests with
coverage" also runs for 3.12; modify the "Run tests" step (named "Run tests") to
skip when matrix.python-version == '3.12' (e.g., add an if condition such as
matrix.python-version != '3.12') so only one of the two steps runs for that
Python version and keep the "Run tests with coverage" step unchanged.

In `@tests/test_validate.py`:
- Around line 44-46: Open JSON files with an explicit encoding by adding
encoding="utf-8" to every open() call in tests/test_validate.py — for example
update the open(...) in test_required_fields and the other test functions (the
open calls around lines 45, 56, 67, 80, 94) so they use open(...,
encoding="utf-8") to match vera_bench/validate.py and avoid platform-dependent
defaults.
- Around line 12-24: Tests define a local REQUIRED_FIELDS list that duplicates
vera_bench.validate.REQUIRED_FIELDS; replace the local list by importing
REQUIRED_FIELDS from vera_bench.validate (e.g. from vera_bench.validate import
REQUIRED_FIELDS) and use that imported symbol in tests (remove the duplicate
definition) so tests stay in sync with the source.

---

Duplicate comments:
In @.github/workflows/ci.yml:
- Around line 25-26: Update the "Install vera compiler" step to install Vera
from an immutable ref instead of the default branch: change the pip install
invocation to reference a specific release tag or commit SHA (e.g., append
@<TAG_OR_SHA> to the git+https URL) so the workflow is reproducible; apply the
same change to the identical "Install vera compiler" step in the test job as
well.

In `@vera_bench/validate.py`:
- Around line 242-246: The aggregation that computes passed currently sums rows
where r["errors"], r["fields_ok"], r["vera_found"], and r["check_pass"] are
checked; update this to also require r["verify_pass"] so passed matches the
corrected ok logic—locate the comprehension assigning to the variable passed in
validate.py and add r["verify_pass"] to the boolean conjunction used for each r
in results.
- Around line 202-206: The overall success calculation omits the verification
result: update the `ok` logic inside the loop over `results` so it also requires
verification to have passed by including `verify_pass` (e.g., check
`r["verify_pass"] is not False` or `r.get("verify_pass") is not False`)
alongside `r["fields_ok"]`, `r["vera_found"]`, `r["check_pass"]`, and `not
r["errors"]`; this will treat a False verification as failure while allowing
None (verification not run) to not force a failure.
- Around line 150-158: The test currently treats a case as passed based only on
stdout comparison; update the logic around runner.run_fn/normalize_output to
first verify the process succeeded: check run.returncode == 0 (and any
runner.timed_out or equivalent flag) before counting a pass, and if non-zero or
timed out append a descriptive error to result["errors"] including return code
and stderr; ensure runner.run_fn is invoked with a timeout per the guideline if
not already and only perform stdout normalization/expected comparison when the
exit code is zero.

In `@vera_bench/vera_runner.py`:
- Around line 69-72: The current parsing concatenates stdout and stderr into
combined and passes that to json.loads which breaks when stderr contains
non-JSON; change the parsing logic in the method that builds combined (and in
verify()) to first attempt json.loads(result.stdout), and only if that raises
JSONDecodeError attempt json.loads(result.stderr) (or raise a clear error if
neither parses), updating usage of the combined variable and the except
JSONDecodeError handler accordingly so stdout is preferred and stderr is a
fallback.

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@vera_bench/validate.py`:
- Around line 108-110: The current logic only trusts check.passed/verify.passed;
update the checks to require explicit zero exit codes as well: for the call
using runner.check(vera_file) ensure you assert both check.exit_code == 0 and
check.passed before setting result["check_pass"], and do the same for
runner.verify (the verify block around lines 124-126) using verify.exit_code ==
0 && verify.passed before setting result["verify_pass"]; if the exit_code is
non-zero, treat it as a failure (set the result flag false) and surface/log the
exit_code or error as appropriate.
- Around line 146-149: The loop over test_cases assumes each tc is a dict and
calls tc.get() outside a try, causing AttributeError on malformed entries;
update the loop in validate.py (the for tc in test_cases block) to first check
isinstance(tc, dict) or wrap tc access in a try/except, and on failure increment
the per-problem error counter (e.g., result["errors"] or record the error for
that problem) and continue; only call tc.get("args", []) and tc.get("expected")
inside the safe block so malformed test cases don't crash the validator and are
recorded as errors.

---

Duplicate comments:
In `@vera_bench/validate.py`:
- Around line 207-210: The overall pass condition currently computes ok =
r["fields_ok"] and r["vera_found"] and r["check_pass"] and not r["errors"] but
misses r["verify_pass"], so update both places where that condition is used (the
loop over results and the later independent check that duplicates this logic) to
include and r["verify_pass"] (i.e., ok = r["fields_ok"] and r["vera_found"] and
r["check_pass"] and r["verify_pass"] and not r["errors"]); ensure you modify the
expressions referencing keys fields_ok, vera_found, check_pass, verify_pass, and
errors so failed verification will no longer return an overall success.