feat: vera test String & Float64 input generation (#169)#441
Conversation
Extend the Z3-guided contract testing engine to generate inputs for String and Float64 parameters, closing issue #169. - vera/smt.py: add declare_string() (z3.String, sequence sort) and declare_float64() (z3.Real, mathematical reals); update _vera_type_to_z3_sort() to return StringSort/RealSort for those types - vera/tester.py: add STRING and FLOAT64 to _Z3_SUPPORTED; add _NEEDS_RAW set; add _BOUNDARY_STRING and _BOUNDARY_FLOAT64 constants; extend _generate_inputs(), _seed_boundaries(), and _extract_values() to handle both types; switch _run_trials() to raw_args calling convention when any param is String/Float64 (required for String is i32_pair WASM ABI); widen all relevant type annotations Float64 uses Z3 RealSort -- NaN, +/-inf, and subnormals are not generated (IEEE 754 special values cannot be represented as Z3 reals). Strings are capped at 50 characters. ADT input generation remains unsupported and is tracked in #440. Documentation updates: SKILL.md, KNOWN_ISSUES.md, ROADMAP.md, HISTORY.md, CHANGELOG.md, README.md, TESTING.md, docs/* regenerated. Version bump: 0.0.105 -> 0.0.106. Co-Authored-By: Claude <noreply@anthropic.invalid>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughv0.0.106 adds Z3-backed input generation for String (bounded sequences ≤50 chars) and Float64 (Z3 Reals with boundary seeding), updates SMT helpers and tester runtime to handle heterogeneous values and raw_args calling convention, and updates tests/docs; ADT input generation remains unsupported. Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Client Code
participant Tester as vera.tester
participant SMT as vera.smt (SmtContext)
participant Z3 as Z3 Solver
Client->>Tester: run vera test (func with String/Float64 params)
Tester->>SMT: declare_string(name) / declare_float64(name)
SMT->>Z3: create z3.String / z3.Real symbols
Z3-->>SMT: SeqRef / ArithRef
Tester->>Z3: add constraints (length ≤ 50, boundary seeds)
Z3-->>Tester: model
Tester->>Tester: extract values (z3.String→str, z3.Real→float)
Tester->>Tester: select calling convention
alt String or Float64 present
Tester->>Client: execute using raw_args (converted strings)
else
Tester->>Client: execute using standard args
end
Client-->>Tester: result
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related issues
Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #441 +/- ##
==========================================
+ Coverage 90.31% 90.32% +0.01%
==========================================
Files 49 49
Lines 19278 19335 +57
Branches 220 220
==========================================
+ Hits 17410 17464 +54
- Misses 1864 1867 +3
Partials 4 4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
vera/smt.py (1)
253-267:⚠️ Potential issue | 🟡 MinorStale docstring: String and Float64 are now supported.
The docstring at line 255 states "Returns None for unsupported types (String, Float64, Unit, TypeVar, function types)" but the implementation now supports String and Float64.
Proposed fix
def _vera_type_to_z3_sort( self, ty: Type, *, self_ref_key: str | None = None, self_ref_dt: Any | None = None, ) -> z3.SortRef | None: """Map a Vera Type to a Z3 sort. - Returns None for unsupported types (String, Float64, Unit, - TypeVar, function types). + Returns None for unsupported types (Unit, TypeVar, function types). """🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@vera/smt.py` around lines 253 - 267, The docstring above the Vera-to-Z3 sort mapping function is stale: it says String and Float64 are unsupported though the code returns z3.StringSort() and z3.RealSort() for ty.name == "String" and "Float64"; update the docstring in that mapping function (the one handling PrimitiveType and returning z3.IntSort(), z3.BoolSort(), z3.StringSort(), z3.RealSort()) to remove String and Float64 from the list of unsupported types (and otherwise accurately list remaining unsupported types such as Unit, TypeVar, and function types).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@HISTORY.md`:
- Line 221: The release count summary in HISTORY.md was not updated after adding
the v0.0.106 row; locate the summary line that currently reads "105 tagged
releases" and change the count to "106 tagged releases" so the aggregate matches
the new v0.0.106 entry (search for the string "105 tagged releases" or the
summary paragraph near the changelog table to find and update it).
In `@ROADMAP.md`:
- Line 230: Update the release count in ROADMAP.md: locate the sentence
containing "**630+ commits, 105 tagged releases, 3,205 tests, 96% coverage, 72
conformance programs, 30 examples, 13 spec chapters.**" and change "105 tagged
releases" to "106 tagged releases" so the document reflects the newly added
v0.0.106 tag.
- Line 11: Update the opening sentence in ROADMAP.md to reflect the current
release version: find the literal version string "v0.0.105" in the file (the
first line/intro sentence) and change it to "v0.0.106" so the roadmap text
matches the PR and metrics; ensure no other occurrences are left inconsistent.
---
Outside diff comments:
In `@vera/smt.py`:
- Around line 253-267: The docstring above the Vera-to-Z3 sort mapping function
is stale: it says String and Float64 are unsupported though the code returns
z3.StringSort() and z3.RealSort() for ty.name == "String" and "Float64"; update
the docstring in that mapping function (the one handling PrimitiveType and
returning z3.IntSort(), z3.BoolSort(), z3.StringSort(), z3.RealSort()) to remove
String and Float64 from the list of unsupported types (and otherwise accurately
list remaining unsupported types such as Unit, TypeVar, and function types).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: f5112a9d-d8aa-40ac-b254-10abaf8a4c9e
⛔ Files ignored due to path filters (6)
docs/SKILL.mdis excluded by!docs/**docs/index.htmlis excluded by!docs/**docs/index.mdis excluded by!docs/**docs/llms-full.txtis excluded by!docs/**docs/llms.txtis excluded by!docs/**docs/sitemap.xmlis excluded by!docs/**
📒 Files selected for processing (14)
CHANGELOG.mdHISTORY.mdKNOWN_ISSUES.mdREADME.mdROADMAP.mdSKILL.mdTESTING.mdpyproject.tomltests/test_tester.pytests/test_tester_coverage.pytests/test_verifier_coverage.pyvera/__init__.pyvera/smt.pyvera/tester.py
VeraBench implicationsThis is significant for the benchmark in several ways: 1. String test_cases may become possibleCurrently 26 of 50 problems have empty Question for the vera compiler team: Does this change also affect 2.
|
- HISTORY.md: 105 -> 106 tagged releases in totals line - ROADMAP.md: v0.0.105 -> v0.0.106 in intro; 105 -> 106 tagged releases in summary line - vera/smt.py: update _vera_type_to_z3_sort() docstring — String and Float64 are now supported (StringSort and RealSort); remaining unsupported types are Unit, TypeVar, and function types Co-Authored-By: Claude <noreply@anthropic.invalid>
|
Thanks for the detailed analysis. Answering your question directly:
What this PR adds is different: On String return values from Summary for vera-bench#25: String inputs to |
Summary
vera test: Float64 and compound type input generation #169)vera test: Float64 and compound type input generation #169Implementation
The key is the dual calling convention in execute(): args (direct list[int|float] to wasmtime) vs raw_args (strings routed through fn_param_types which handles i32_pair for String via _alloc_string_arg()). The raw_args path was designed for this use case.
vera/smt.py: Added declare_string() and declare_float64(); updated _vera_type_to_z3_sort() to return StringSort/RealSort.
vera/tester.py: Extended _Z3_SUPPORTED, added _NEEDS_RAW, new boundary constants, new branches in _generate_inputs() / _seed_boundaries() / _extract_values(), switched _run_trials() to raw_args for String/Float64 params. Widened return types throughout.
Test plan
Generated with Claude Code
Summary by CodeRabbit
New Features
vera testnow generates inputs for String and Float64 parameters (previously skipped).Documentation