Skip to content

feat: vera test String & Float64 input generation (#169)#441

Merged
aallan merged 2 commits into
mainfrom
feat/test-string-float64-169
Mar 31, 2026
Merged

feat: vera test String & Float64 input generation (#169)#441
aallan merged 2 commits into
mainfrom
feat/test-string-float64-169

Conversation

@aallan

@aallan aallan commented Mar 31, 2026

Copy link
Copy Markdown
Owner

Summary

  • vera test now generates Z3 inputs for String and Float64 parameters, removing the long-standing SKIPPED limitation for those types (vera test: Float64 and compound type input generation #169)
  • String uses Z3 sequence sort with a 50-char length cap; boundaries include empty string, single char, whitespace strings
  • Float64 uses Z3 mathematical real sort; boundaries include 0.0, +/-1.0, +/-0.5, +/-10.0, +/-1e10, +/-1e-10
  • Both types route through the raw_args calling convention -- required because String is i32_pair in WASM ABI (pointer + length)
  • IEEE 754 special values (NaN, +/-inf, subnormals) are not generated -- Z3 RealSort is mathematical reals, not IEEE 754
  • ADT input generation remains unsupported -- tracked in vera test: ADT parameter input generation #440
  • 20 files changed; 8 new tests; version bump 0.0.105 -> 0.0.106; closes vera test: Float64 and compound type input generation #169

Implementation

The key is the dual calling convention in execute(): args (direct list[int|float] to wasmtime) vs raw_args (strings routed through fn_param_types which handles i32_pair for String via _alloc_string_arg()). The raw_args path was designed for this use case.

vera/smt.py: Added declare_string() and declare_float64(); updated _vera_type_to_z3_sort() to return StringSort/RealSort.

vera/tester.py: Extended _Z3_SUPPORTED, added _NEEDS_RAW, new boundary constants, new branches in _generate_inputs() / _seed_boundaries() / _extract_values(), switched _run_trials() to raw_args for String/Float64 params. Widened return types throughout.

Test plan

  • pytest tests/test_tester.py tests/test_tester_coverage.py tests/test_verifier_coverage.py -- all 62 tests pass
  • mypy vera/ -- clean
  • Full pytest tests/ -- 3205 tests, 0 failures
  • All 72 conformance programs pass
  • check_doc_counts / check_version_sync / check_limitations_sync -- all consistent

Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • vera test now generates inputs for String and Float64 parameters (previously skipped).
    • String inputs limited to 50 characters; Float64 generation seeded at boundaries and excludes NaN, ±∞, and subnormals.
    • Calling convention updated so String/Float64 arguments are handled correctly at runtime.
  • Documentation

    • Updated project status, changelog, roadmap, HISTORY, and known limitations.
    • ADT parameters remain unsupported.

Extend the Z3-guided contract testing engine to generate inputs for
String and Float64 parameters, closing issue #169.

- vera/smt.py: add declare_string() (z3.String, sequence sort) and
  declare_float64() (z3.Real, mathematical reals); update
  _vera_type_to_z3_sort() to return StringSort/RealSort for those types
- vera/tester.py: add STRING and FLOAT64 to _Z3_SUPPORTED; add
  _NEEDS_RAW set; add _BOUNDARY_STRING and _BOUNDARY_FLOAT64 constants;
  extend _generate_inputs(), _seed_boundaries(), and _extract_values()
  to handle both types; switch _run_trials() to raw_args calling
  convention when any param is String/Float64 (required for String is
  i32_pair WASM ABI); widen all relevant type annotations

Float64 uses Z3 RealSort -- NaN, +/-inf, and subnormals are not
generated (IEEE 754 special values cannot be represented as Z3 reals).
Strings are capped at 50 characters. ADT input generation remains
unsupported and is tracked in #440.

Documentation updates: SKILL.md, KNOWN_ISSUES.md, ROADMAP.md,
HISTORY.md, CHANGELOG.md, README.md, TESTING.md, docs/* regenerated.
Version bump: 0.0.105 -> 0.0.106.

Co-Authored-By: Claude <noreply@anthropic.invalid>
@coderabbitai

coderabbitai Bot commented Mar 31, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 7e64faf2-ec44-4c04-864f-5de1cc91cae8

📥 Commits

Reviewing files that changed from the base of the PR and between b7e3cb3 and 1177385.

📒 Files selected for processing (3)
  • HISTORY.md
  • ROADMAP.md
  • vera/smt.py

📝 Walkthrough

Walkthrough

v0.0.106 adds Z3-backed input generation for String (bounded sequences ≤50 chars) and Float64 (Z3 Reals with boundary seeding), updates SMT helpers and tester runtime to handle heterogeneous values and raw_args calling convention, and updates tests/docs; ADT input generation remains unsupported.

Changes

Cohort / File(s) Summary
Documentation & versioning
CHANGELOG.md, HISTORY.md, README.md, pyproject.toml, vera/__init__.py
Bumped release to 0.0.106 and documented String/Float64 Z3 input generation (≤50 chars, real sort + boundary seeds).
Limitations & roadmap
KNOWN_ISSUES.md, ROADMAP.md, SKILL.md, TESTING.md
Removed String/Float64 from unsupported list; marked ADT inputs as remaining limitation (#440). Incremented test and release counters.
SMT context
vera/smt.py
Added SmtContext.declare_string and declare_float64; _vera_type_to_z3_sort now maps Stringz3.StringSort(), Float64z3.RealSort().
Tester & runtime
vera/tester.py
Widened value types to `int
Tests
tests/test_tester.py, tests/test_tester_coverage.py, tests/test_verifier_coverage.py
Reclassified String/Float64 cases from skipped→tested, added coverage tests for String sort mapping, renamed and added tests exercising Z3 generation, and made ADT the canonical unsupported case.

Sequence Diagram(s)

sequenceDiagram
    participant Client as Client Code
    participant Tester as vera.tester
    participant SMT as vera.smt (SmtContext)
    participant Z3 as Z3 Solver

    Client->>Tester: run vera test (func with String/Float64 params)
    Tester->>SMT: declare_string(name) / declare_float64(name)
    SMT->>Z3: create z3.String / z3.Real symbols
    Z3-->>SMT: SeqRef / ArithRef
    Tester->>Z3: add constraints (length ≤ 50, boundary seeds)
    Z3-->>Tester: model
    Tester->>Tester: extract values (z3.String→str, z3.Real→float)
    Tester->>Tester: select calling convention
    alt String or Float64 present
        Tester->>Client: execute using raw_args (converted strings)
    else
        Tester->>Client: execute using standard args
    end
    Client-->>Tester: result
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

Possibly related PRs

Suggested labels

compiler, tests, docs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main change: adding String and Float64 input generation to vera test, with a direct reference to the driving issue (#169).
Linked Issues check ✅ Passed The PR successfully implements Z3-based input generation for Float64 (via RealSort with boundary seeding) and String (via StringSort with 50-character cap), meeting the primary coding requirements from #169. ADT generation remains unsupported and tracked separately in #440, as documented.
Out of Scope Changes check ✅ Passed All changes are in scope: implementation of Float64/String generation in vera/smt.py and vera/tester.py, type annotation updates to support the new types, version bumps, and comprehensive test coverage updates. No extraneous refactoring or unrelated modifications present.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/test-string-float64-169

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov

codecov Bot commented Mar 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.10145% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.32%. Comparing base (3939b9f) to head (1177385).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
vera/tester.py 96.36% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #441      +/-   ##
==========================================
+ Coverage   90.31%   90.32%   +0.01%     
==========================================
  Files          49       49              
  Lines       19278    19335      +57     
  Branches      220      220              
==========================================
+ Hits        17410    17464      +54     
- Misses       1864     1867       +3     
  Partials        4        4              
Flag Coverage Δ
javascript 50.56% <ø> (ø)
python 95.27% <97.10%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
vera/smt.py (1)

253-267: ⚠️ Potential issue | 🟡 Minor

Stale docstring: String and Float64 are now supported.

The docstring at line 255 states "Returns None for unsupported types (String, Float64, Unit, TypeVar, function types)" but the implementation now supports String and Float64.

Proposed fix
     def _vera_type_to_z3_sort(
         self,
         ty: Type,
         *,
         self_ref_key: str | None = None,
         self_ref_dt: Any | None = None,
     ) -> z3.SortRef | None:
         """Map a Vera Type to a Z3 sort.

-        Returns None for unsupported types (String, Float64, Unit,
-        TypeVar, function types).
+        Returns None for unsupported types (Unit, TypeVar, function types).
         """
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@vera/smt.py` around lines 253 - 267, The docstring above the Vera-to-Z3 sort
mapping function is stale: it says String and Float64 are unsupported though the
code returns z3.StringSort() and z3.RealSort() for ty.name == "String" and
"Float64"; update the docstring in that mapping function (the one handling
PrimitiveType and returning z3.IntSort(), z3.BoolSort(), z3.StringSort(),
z3.RealSort()) to remove String and Float64 from the list of unsupported types
(and otherwise accurately list remaining unsupported types such as Unit,
TypeVar, and function types).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@HISTORY.md`:
- Line 221: The release count summary in HISTORY.md was not updated after adding
the v0.0.106 row; locate the summary line that currently reads "105 tagged
releases" and change the count to "106 tagged releases" so the aggregate matches
the new v0.0.106 entry (search for the string "105 tagged releases" or the
summary paragraph near the changelog table to find and update it).

In `@ROADMAP.md`:
- Line 230: Update the release count in ROADMAP.md: locate the sentence
containing "**630+ commits, 105 tagged releases, 3,205 tests, 96% coverage, 72
conformance programs, 30 examples, 13 spec chapters.**" and change "105 tagged
releases" to "106 tagged releases" so the document reflects the newly added
v0.0.106 tag.
- Line 11: Update the opening sentence in ROADMAP.md to reflect the current
release version: find the literal version string "v0.0.105" in the file (the
first line/intro sentence) and change it to "v0.0.106" so the roadmap text
matches the PR and metrics; ensure no other occurrences are left inconsistent.

---

Outside diff comments:
In `@vera/smt.py`:
- Around line 253-267: The docstring above the Vera-to-Z3 sort mapping function
is stale: it says String and Float64 are unsupported though the code returns
z3.StringSort() and z3.RealSort() for ty.name == "String" and "Float64"; update
the docstring in that mapping function (the one handling PrimitiveType and
returning z3.IntSort(), z3.BoolSort(), z3.StringSort(), z3.RealSort()) to remove
String and Float64 from the list of unsupported types (and otherwise accurately
list remaining unsupported types such as Unit, TypeVar, and function types).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: f5112a9d-d8aa-40ac-b254-10abaf8a4c9e

📥 Commits

Reviewing files that changed from the base of the PR and between 3939b9f and b7e3cb3.

⛔ Files ignored due to path filters (6)
  • docs/SKILL.md is excluded by !docs/**
  • docs/index.html is excluded by !docs/**
  • docs/index.md is excluded by !docs/**
  • docs/llms-full.txt is excluded by !docs/**
  • docs/llms.txt is excluded by !docs/**
  • docs/sitemap.xml is excluded by !docs/**
📒 Files selected for processing (14)
  • CHANGELOG.md
  • HISTORY.md
  • KNOWN_ISSUES.md
  • README.md
  • ROADMAP.md
  • SKILL.md
  • TESTING.md
  • pyproject.toml
  • tests/test_tester.py
  • tests/test_tester_coverage.py
  • tests/test_verifier_coverage.py
  • vera/__init__.py
  • vera/smt.py
  • vera/tester.py

Comment thread HISTORY.md
Comment thread ROADMAP.md
Comment thread ROADMAP.md Outdated
@aallan

aallan commented Mar 31, 2026

Copy link
Copy Markdown
Owner Author

VeraBench implications

This is significant for the benchmark in several ways:

1. String test_cases may become possible

Currently 26 of 50 problems have empty test_cases — most of these are Tier 2 (string/array) problems where vera run --fn couldn't handle string arguments (vera-bench#25 constraint comment, vera#403). If the raw_args calling convention now supports strings at runtime (not just in vera test), we could add string test_cases to Tier 2 problems and get run_correct metrics for them.

Question for the vera compiler team: Does this change also affect vera run --fn argument passing, or is it limited to vera test? Specifically, does vera run --fn greet -- "Alice" now work? If so, please comment on vera-bench#25 so we can plan adding string test_cases to Tier 2 problems.

2. vera test as additional validation

We could integrate vera test into the VeraBench validation pipeline for problems with String/Float64 parameters. Related: vera-bench#25 (expand to 75+ problems) — new problems could use vera test as an additional correctness check during development. Also relates to vera-bench#31 (automated scheduled runs) where vera test could be part of the CI validation step.

3. Future problem expansion

For vera-bench#25, String and Float64 input generation means new Tier 2 problems can be more thoroughly tested. Float64 problems would be a new problem category. The vera-bench#25 constraints about vera test limitations (previously noted via vera#169) are now partially lifted.

4. Still blocked for ADTs

Tier 3/4 ADT problems remain untestable via vera test (vera#440). The empty test_cases for List/Tree/Option problems will persist until ADT input generation lands.

Related vera-bench issues

  • vera-bench#25 — Expand to 75+ problems (String/Float64 test generation unblocks new problem types)
  • vera-bench#31 — Automated scheduled runs (vera test could be part of validation)
  • vera-bench#5 — Test coverage (integrating vera test into the validation pipeline)

- HISTORY.md: 105 -> 106 tagged releases in totals line
- ROADMAP.md: v0.0.105 -> v0.0.106 in intro; 105 -> 106 tagged releases
  in summary line
- vera/smt.py: update _vera_type_to_z3_sort() docstring — String and
  Float64 are now supported (StringSort and RealSort); remaining
  unsupported types are Unit, TypeVar, and function types

Co-Authored-By: Claude <noreply@anthropic.invalid>
@aallan

aallan commented Mar 31, 2026

Copy link
Copy Markdown
Owner Author

Thanks for the detailed analysis. Answering your question directly:

vera run --fn string argument support is NOT new to this PR. String argument passing (allocating strings into WASM linear memory for vera run --fn greet -- "Alice") was implemented in v0.0.102 as part of #263/#403 — the CompileResult.fn_param_types field and the raw_args path in execute() that handles i32_pair. That has been working since then.

What this PR adds is different: vera test can now generate String and Float64 values via Z3 and feed them to the contract testing engine. Previously, functions with those parameter types were skipped entirely by the Z3 input generation pipeline.

On String return values from vera run --fn: If a function returns a String, vera run --fn currently prints the WASM pointer integer (e.g. 8219), not the decoded string. This is a limitation of the CLI output layer — String return values are i32_pair in WASM and the result printer does not decode them back to text. For VeraBench test_cases that check String return values, you would need the function to use IO.print instead of returning a String directly.

Summary for vera-bench#25: String inputs to vera run --fn work (since v0.0.102). String return values from vera run --fn do not print as text (separate issue). vera test now validates String/Float64 contracts, which is useful for contract correctness checking in the benchmark pipeline but not for output comparison in test_cases.

@aallan aallan merged commit cc51d1b into main Mar 31, 2026
19 checks passed
@aallan aallan deleted the feat/test-string-float64-169 branch March 31, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vera test: Float64 and compound type input generation

1 participant