Skip to content

v0.0.134 — Active reclamation of host-store handles for Map/Set/Decimal (closes #573, #575, #576)#577

Merged
aallan merged 9 commits into
mainfrom
claude/fix-573-handle-reclamation
May 6, 2026
Merged

v0.0.134 — Active reclamation of host-store handles for Map/Set/Decimal (closes #573, #575, #576)#577
aallan merged 9 commits into
mainfrom
claude/fix-573-handle-reclamation

Conversation

@aallan

@aallan aallan commented May 6, 2026

Copy link
Copy Markdown
Owner

Summary

Closes the entire host-handle leak class in one PR via heap-wrap-as-ADT migration for Map, Set, and Decimal.

  • Pre-fix: every map_new / set_add / decimal_add / etc. allocated a fresh entry in the corresponding Python-side store and never released transient predecessors. A 10 000-iteration array_fold over map_insert left 10 001 entries in _map_store at exit. Bounded by Python GC at process exit, so single-shot programs were unaffected, but long-running execution contexts (server programs, repeated execute()) leaked monotonically across all three types.
  • Post-fix: every Map<K, V> / Set<T> / Decimal value is a pointer to an 8-byte wrapper ADT on the GC heap. A new 64 KiB wrap-table region in linear memory tracks every wrapper allocation; Phase 2c of $gc_collect walks the wrap table and fires host_decref_handle(kind, handle) for every unmarked wrapper, evicting the corresponding store entry. Survivors compact in place.

Closes

Closes #573.
Closes #575.
Closes #576.
Closes #579.

What changed

File Change
vera/codegen/assembly.py New wrap-table region (gated on _needs_wrap_table), $register_wrapper helper, Phase 2c walk + compaction in $gc_collect, host_decref_handle import, register_wrapper export. Flag flips on any of _map_ops_used / _set_ops_used / _decimal_ops_used.
vera/codegen/api.py host_decref_handle(kind, handle) Python dispatcher (kind=1 Map, 2 Set, 3 Decimal). New _wrap_handle(caller, kind, raw) generic helper. _alloc_map_wrapper(caller, dict) for JSON/HTML internal Map allocations. host_decimal_from_string and host_decimal_div wrap their result handles before stuffing into Some.
vera/wasm/calls_containers.py All call-sites wrap on construction, unwrap on consumption. 8 Map ops + 6 Set ops + 10 Decimal ops. _emit_wrap_handle shadow-pushes the new wrapper after construction (matches ADT-constructor pattern in data.py) so nested expressions like decimal_add(decimal_from_int(a), decimal_from_int(b)) are GC-safe.
vera/wasm/helpers.py _HOST_HANDLE_TYPES is now empty. All three types are wrapper-pointer-rooted.
vera/wasm/json_serde.py / html_serde.py write_* use the wrapping map_alloc(caller, dict) signature; read_* unwrap before host-store lookup. Required so json_has_field / html_get_attr (which dispatch to map_contains on JObject / HtmlElement Map fields) work post-Map-migration.
vera/browser/runtime.mjs Mirror: host_decref_handle JS dispatcher (all three kinds), wrapHandle(kind, raw), JObject + HtmlElement wrap on write / unwrap on read, decimal_from_string + decimal_div wrap result before Some.
tests/test_codegen.py TestHostHandleReclamation573 (7 stress / regression tests). TestOpaqueHandleParamRooting347 Map/Set/Decimal tests flipped from must-NOT-be-rooted to MUST-be-rooted-after-#573. TestArrayFoldHandleRooting490 helper renamed and assertion flipped.
ExecuteResult.host_store_sizes New field exposing post-execution store population so tests can verify reclamation without linker introspection.

Test plan

  • pytest tests/ -q — 3,713 passed, 14 skipped (was 3,706; +7 new tests in TestHostHandleReclamation573)
  • pytest tests/test_browser.py -q — 99 passed (browser parity)
  • mypy vera/ — clean
  • python scripts/check_conformance.py — 82 / 82
  • python scripts/check_examples.py — 33 / 33
  • All sync scripts (version, doc counts, limitations, site assets, allowlists) pass
  • Pre-commit hooks pass

Stress test results (the proof)

Test Pre-fix store size Post-fix store size
10 000-iter map_insert chain 10 001 1
5 000 transient JObject parses ~5 000 < 1 500
10 000-iter set_add chain 10 001 < 2 000
5 000-iter decimal_add chain ~10 000 < 1 500

Steady-state residual is bounded by GC cadence rather than total allocations — Set's host op is cheaper than Map's so GC fires less often, hence the slightly higher residual; Decimal allocates ~2 entries per iteration (accumulator + per-iteration from_int) plus the final decimal_eq pair.

Notes for reviewers

Why all three in one PR. The mechanism (wrap-table + Phase 2c sweep + destructor host import) is the design risk; once Map proved it works, Set and Decimal are mechanical mirrors of the same pattern. Three commits in one PR keeps each commit independently bisectable, gives one design conversation instead of three.

The shadow-push fix in _emit_wrap_handle was found via the Decimal stress tests — the original design had wrappers sitting on the operand stack between sub-expressions without GC visibility, which broke nested calls like decimal_add(decimal_from_int(a), decimal_from_int(b)) if GC fired during the second decimal_from_int's allocation. The fix matches the existing data.py ADT-constructor pattern.

JSON/HTML internal Map wrapping was a forced dependency of the user-facing Map migration. JObject's Map<String, Json> field and HtmlElement's Map<String, String> attrs are both consumed by user code via map_get / map_contains (through the prelude's json_has_field / html_get_attr); once those operations expect wrapper pointers, the host-side parsers had to wrap their internal Map allocations to maintain type compatibility. Not a separate issue, but worth flagging in review.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Host-side wrapper ADT support for Map/Set/Decimal to unify cross-boundary handle management and marshalling.
  • Bug Fixes

    • Improved host-store reclamation and GC integration for Map/Set/Decimal with safer decref/unwrapping semantics.
  • Documentation

    • Updated changelog, history, roadmap, README, KNOWN_ISSUES and testing docs for v0.0.134 and revised metrics.
  • Tests

    • Added extensive regression tests covering post-migration rooting, shadow-push behaviour and host-store reclamation.
  • Chores

    • Bumped project version to v0.0.134.

…-as-ADT (closes #573 phase 1)

Pre-fix every map_new / map_insert / map_remove allocated a new
entry in `_map_store` (Python-side dict in vera/codegen/api.py)
without releasing transient predecessors.  A 10 000-iteration
array_fold over map_insert left 10 001 entries at execute() exit;
bounded by Python GC at process exit so single-shot programs were
unaffected, but long-running execution contexts leaked monotonically.

Post-fix the heap-wrap-as-ADT design from the issue body:

* Every Map<K, V> value is a pointer to an 8-byte wrapper ADT on
  the GC heap (tag at offset 0, raw host handle at offset 4).

* New 64 KiB wrap-table region in linear memory tracks every
  wrapper allocation as (ptr, kind, handle) entries.
  $register_wrapper appends; sweep walks and compacts.

* Phase 2c of $gc_collect — a new pass between mark and sweep —
  walks the wrap table.  Marked entries are kept (compacted in
  place).  Unmarked entries fire host_decref_handle(kind, handle)
  which evicts the corresponding entry from _map_store.

* Every Map call-site in vera/wasm/calls_containers.py wraps on
  construction (new/insert/remove) and unwraps on consumption
  (get/contains/size/keys/values) via i32.load offset=4.

* JSON's JObject and HTML's HtmlElement attrs use the same
  wrapper path via _alloc_map_wrapper in api.py — the prelude's
  json_has_field dispatches to map_contains on JObject's Map
  field, so the field MUST be a wrapper pointer for unwrap to
  produce the correct raw handle.

* `Map` removed from _HOST_HANDLE_TYPES — wrapper pointers ARE
  Vera-heap pointers and need shadow-stack rooting across
  allocating calls.  Set / Decimal stay in the set until their
  own follow-ups (#575, #576).

* Browser parity in vera/browser/runtime.mjs: host_decref_handle,
  allocMapWrapper, JObject + HtmlElement wrap on write, unwrap
  on read.

* ExecuteResult.host_store_sizes — new field exposing post-
  execution store population so tests can verify reclamation
  without invasive linker introspection.

Three regression tests in TestHostHandleReclamation573:

* 10 000-iteration map_insert chain via array_fold:
  _map_store has 1 entry at exit (was 10 001).
* 5 000 transient JObject parses via array_fold:
  _map_store < 1 500 (was monotonic in iterations).
* Functional integrity: map_get returns correct values across
  heavy GC pressure.

Plus the post-#573 expectation flip in
TestOpaqueHandleParamRooting347::test_map_param_shadow_pushed_after_573:
Map params now MUST appear in shadow_push (was: must not).
Set / Decimal pinning tests unchanged.

Phase 2 (Set, #575) and phase 3 (Decimal, #576) are mechanical
mirrors filed as separate follow-ups.

Co-Authored-By: Claude <noreply@anthropic.invalid>
@coderabbitai

coderabbitai Bot commented May 6, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds heap-wrap-as-ADT support and wrap-table GC integration so Map/Set/Decimal host handles become GC-managed wrapper ADTs; per-call unwrap/wrap wiring, browser/runtime and JSON/HTML marshalling updates, assembly emits wrap-table and Phase 2c to call back to host_decref_handle, and tests/docs/version bumped to v0.0.134.

Changes

Heap‑wrap‑as‑ADT + Wrap‑Table GC Integration

Layer / File(s) Summary
Release / Version
pyproject.toml, vera/__init__.py, CHANGELOG.md, HISTORY.md, README.md, ROADMAP.md, TESTING.md
Bump to v0.0.134; add release notes, update changelogs, history, README, roadmap, testing metrics and diff anchors.
Data Shape & Introspection
vera/codegen/api.py
Add ExecuteResult.host_store_sizes; register _host_store_refs for map,set,decimal; add wrapper-layout constants and helpers (_wrap_handle, _alloc_map_wrapper, _call_register_wrapper) and expose host_decref_handle via linker when needed.
Feature Flagging
vera/codegen/core.py, vera/wasm/context.py
Introduce internal _needs_wrap_table: bool = False gating flag with explanatory comments (no runtime behaviour change yet).
Host‑handle Classification
vera/wasm/helpers.py
Remove Map/Set/Decimal from _HOST_HANDLE_TYPES (now empty) and update _is_host_handle_type commentary/behaviour for the wrap‑as‑ADT migration.
WASM Call Translation (wrap/unwrap)
vera/wasm/calls_containers.py
Add per-call unwrap/wrap semantics and wrapper registration for Map ops and extend wrapper handling to Set and Decimal operations (unwrap inputs, wrap results where required).
Marshalling: JSON / HTML
vera/wasm/json_serde.py, vera/wasm/html_serde.py
Change MapAllocFn to accept wasmtime.Caller; write_* paths now allocate wrapper pointers (wrapper_ptr) via host alloc wrappers instead of raw handles; read_* unwrap wrapper_ptr (handle at wrapper_ptr + 4) and validate via host stores.
Browser Runtime
vera/browser/runtime.mjs
Add allocMapWrapper, _MAP_HANDLE_TAG, host_decref_handle and switch JSON/HTML marshalling to wrapper-backed allocations; update read/unwrapping logic.
Assembly / GC Integration
vera/codegen/assembly.py
When wrap-table is enabled, add wrap-table region globals (gc_wrap_base, gc_wrap_ptr, gc_wrap_end), emit register_wrapper export, and inject Phase 2c GC logic to walk/compact wrap-table and call host_decref_handle for unmarked entries; integrate into _emit_gc_collect.
WASM helpers / rooting
vera/wasm/helpers.py
Adjust host-handle classification comments and logic to reflect Map/Set/Decimal as heap pointers requiring rooting.
Tests
tests/test_codegen.py
Large additions and refactors: add positive post-#573 shadow-rooting tests for Map/Set/Decimal, introduce _assert_handle_extra_rooted_after_573, and add TestHostHandleReclamation573 with comprehensive reclamation/GC tests.
Docs / Known Issues
KNOWN_ISSUES.md
Replace single opaque-store bug entry with distinct Set/Decimal entries referencing issues #575 and #576.

Sequence Diagram

sequenceDiagram
    autonumber
    participant Vera as Vera codegen
    participant WASM as WASM module
    participant Host as Python host runtime
    participant GC as GC collector
    participant Store as Host stores

    Vera->>WASM: emit call for map_new()
    WASM->>Host: host.map_new() (host import)
    Host->>Store: allocate raw i32 handle
    Host->>Host: allocMapWrapper(handle) -> wrapper_ptr
    Host-->>WASM: return wrapper_ptr
    WASM-->>Vera: wrapper_ptr (Vera-heap ADT)

    Vera->>WASM: map_insert(wrapper_ptr, k, v)
    WASM->>WASM: unwrap wrapper_ptr -> raw_handle
    WASM->>Host: host.map_insert(raw_handle, k, v)
    Host->>Store: update store (maybe new raw handle)
    Host->>Host: allocMapWrapper(new_handle) -> new_wrapper_ptr
    Host-->>WASM: return new_wrapper_ptr

    Vera->>GC: gc_collect() safepoint
    GC->>GC: mark reachable Vera heap (wrappers marked if live)
    GC->>WASM: Phase 2c: walk wrap-table for unmarked wrappers
    WASM->>Host: host_decref_handle(kind, handle) for unreachable entries
    Host->>Store: evict unreachable entries from host stores
    GC->>GC: sweep/reclaim unreachable wrapper ADTs
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

Possibly related PRs

Suggested labels

compiler, tests, docs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed Title accurately describes the primary change: version bump to 0.0.134 implementing active reclamation of host-store handles (Map/Set/Decimal) via heap-wrap-as-ADT with GC Phase 2c.
Linked Issues check ✅ Passed All linked objectives (#573, #575, #576) met: Map/Set/Decimal migrated to heap-wrap-as-ADT with Phase 2c GC sweep, host_decref_handle dispatcher, wrap/unwrap throughout wasm layer, browser runtime parity, removed from _HOST_HANDLE_TYPES, extensive regression tests added.
Out of Scope Changes check ✅ Passed All changes are scoped to #573, #575, #576 objectives: wrap-table infra, ADT wrappers, Phase 2c, host_decref_handle, wrap/unwrap logic, browser parity, docs, and regression tests. No unrelated modifications detected.
Docstring Coverage ✅ Passed Docstring coverage is 82.50% which is sufficient. The required threshold is 80.00%.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/fix-573-handle-reclamation

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov

codecov Bot commented May 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.28571% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.98%. Comparing base (bde6611) to head (20d4f77).

Files with missing lines Patch % Lines
vera/browser/runtime.mjs 65.33% 26 Missing ⚠️
vera/wasm/calls_containers.py 95.23% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #577      +/-   ##
==========================================
+ Coverage   90.97%   90.98%   +0.01%     
==========================================
  Files          59       59              
  Lines       22620    22884     +264     
  Branches      259      259              
==========================================
+ Hits        20578    20821     +243     
- Misses       2035     2056      +21     
  Partials        7        7              
Flag Coverage Δ
javascript 57.36% <65.33%> (+0.53%) ⬆️
python 94.82% <96.99%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@HISTORY.md`:
- Around line 301-311: Update the documented test and tag counts to match actual
repository state: run pytest --collect-only -q and capture the real collection
numbers (1,608 tests and 15 collection errors) and update TESTING.md accordingly
(this is the single source of truth used by scripts/check_doc_counts.py), then
propagate the corrected test count into HISTORY.md (replace "3,709" with 1,608
and note the 15 collection errors) and ROADMAP.md so all three files agree; also
remove or correct the "134 tagged releases" claim in HISTORY.md and ROADMAP.md
to reflect the current git tag count (0) and add a TODO/note that release
v0.0.134 must not be assumed until tags are reconciled and an explicit tagging
step is performed.

In `@tests/test_codegen.py`:
- Around line 15886-15899: The test currently allows a large residual in
exec_result.host_store_sizes["map"] by asserting <=100; tighten the reclamation
check to the acceptance bar by either invoking a final GC/collection step before
reading exec_result.host_store_sizes (so Phase 2c destructor hooks run) or
reduce the assertion bound to a single-digit residual (e.g., <=9) to enforce
"only the live final map" behavior; update the assertions in tests referencing
exec_result.host_store_sizes.get("map", 0) (also the similar block around the
second occurrence) and ensure any helper that triggers Phase 2c
(host_decref_handle / destructors) is called prior to sampling.
- Around line 15535-15548: The test's regex only checks that a value is stored
at gc_sp but not that gc_sp is advanced, so update the assertion in
tests/test_codegen.py (the push_pattern used to validate $lookup_or_zero /
gc_shadow_push) to match the full shadow-push sequence: ensure the pattern looks
for the initial "global.get $gc_sp ... local.get (?:0\b|\$p?0\b) ... i32.store"
followed later by "global.get $gc_sp\s+i32.const 4\s+i32.add\s+global.set
$gc_sp" (or combine into a single regex that requires those instructions in
order) so the test fails if the stack pointer is not incremented.

In `@vera/codegen/api.py`:
- Around line 1694-1722: The host_decref_handle import is only defined when
result.map_ops_used is true, but json_parse/html_parse/html_query also allocate
Map wrappers via _alloc_map_wrapper; change the guard so host_decref_handle is
defined under the same predicate used to create/enable
_map_store/_alloc_map_wrapper (i.e. when Map OR Json OR Html usage is present).
Locate the current if result.map_ops_used: block around host_decref_handle and
replace the condition with the combined predicate used for map-wrapper
allocation (e.g. result.map_ops_used or result.json_used or result.html_used, or
the shared helper/flag that _alloc_map_wrapper/_map_store checks) so
json/html-only modules get the destructor import too.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 248faff9-b09d-42f4-9f1b-79c3d528fb36

📥 Commits

Reviewing files that changed from the base of the PR and between bde6611 and 69face3.

⛔ Files ignored due to path filters (6)
  • docs/index.html is excluded by !docs/**
  • docs/index.md is excluded by !docs/**
  • docs/llms-full.txt is excluded by !docs/**
  • docs/llms.txt is excluded by !docs/**
  • docs/sitemap.xml is excluded by !docs/**
  • uv.lock is excluded by !**/*.lock, !uv.lock
📒 Files selected for processing (18)
  • CHANGELOG.md
  • HISTORY.md
  • KNOWN_ISSUES.md
  • README.md
  • ROADMAP.md
  • TESTING.md
  • pyproject.toml
  • tests/test_codegen.py
  • vera/__init__.py
  • vera/browser/runtime.mjs
  • vera/codegen/api.py
  • vera/codegen/assembly.py
  • vera/codegen/core.py
  • vera/wasm/calls_containers.py
  • vera/wasm/context.py
  • vera/wasm/helpers.py
  • vera/wasm/html_serde.py
  • vera/wasm/json_serde.py

Comment thread HISTORY.md
Comment thread tests/test_codegen.py
Comment thread tests/test_codegen.py
Comment thread vera/codegen/api.py Outdated
Comment thread vera/codegen/api.py Outdated
After Map's heap-wrap-as-ADT migration de-risked the design, the
remaining Set and Decimal migrations are mechanical mirrors —
same wrap-table, same destructor host import, same Phase 2c
sweep, only the call-site translators and the ``kind``
discriminator change.  Doing all three in one PR keeps each
commit independently bisectable, gives reviewers one design
conversation instead of three, and avoids two more release
cycles for clearly-mechanical work.

Set migration (closes #575):
- vera/wasm/calls_containers.py — wrap on set_new / set_add /
  set_remove; unwrap on set_contains / set_size / set_to_array.
- vera/codegen/api.py — host_decref_handle now dispatches
  kind=2 to _set_store.pop.
- vera/browser/runtime.mjs — same JS dispatch.

Decimal migration (closes #576):
- vera/wasm/calls_containers.py — wrap on Decimal-returning
  ops (decimal_from_int / _from_float / _neg / _add / _sub /
  _mul / _round); unwrap on Decimal-consuming ops
  (_to_string / _to_float / _eq / _compare).
  decimal_from_string and decimal_div construct Option<Decimal>
  host-side; their inner Decimal handle is wrapped via
  _wrap_handle (new helper in api.py) before Some.
- vera/codegen/api.py — _wrap_handle(caller, kind, raw_handle)
  generic wrapper helper, factored out of _alloc_map_wrapper.
  host_decref_handle dispatches kind=3 to _decimal_store.pop.
- vera/browser/runtime.mjs — wrapHandle JS counterpart, used
  in decimal_from_string and decimal_div.

Correctness fix found while writing stress tests:
- vera/wasm/calls_containers.py::_emit_wrap_handle — now
  shadow-pushes the wrapper pointer right after construction,
  matching the existing ADT-constructor pattern in data.py.
  Without this, nested expressions like
  decimal_add(decimal_from_int(a), decimal_from_int(b)) were
  unsafe — the inner wrapper sits on the operand stack while
  the second decimal_from_int invokes $alloc, and a GC fire
  there would sweep the unmarked wrapper.  Affects all three
  types; the function epilogue's $gc_sp restore clears these
  per-call pushes so the shadow stack doesn't grow unbounded
  across iterations of an enclosing array_fold.

GC-rooting hygiene:
- vera/wasm/helpers.py — _HOST_HANDLE_TYPES is now empty (was
  {Map, Set, Decimal}).  All three are real Vera-heap pointers
  post-#573 and MUST be shadow-stack-rooted.

Test updates:
- tests/test_codegen.py — TestOpaqueHandleParamRooting347
  Set + Decimal tests flipped from "must NOT be shadow-pushed"
  to "MUST be shadow-pushed after #573".
  TestArrayFoldHandleRooting490 helper renamed to
  _assert_handle_extra_rooted_after_573 with strictly-greater
  assertion (Decimal accumulators / elements MUST add
  per-iteration root pushes).
  TestHostHandleReclamation573 grew from 3 to 7 stress tests:
  * test_set_chain_reclaims_transients (10K set_add chain,
    < 2000 entries at exit)
  * test_set_value_correct_after_gc_pressure (1K set_add +
    multiple lookups + size, all on live Set)
  * test_decimal_chain_reclaims_transients (5K decimal_add
    chain, < 1500 entries at exit)
  * test_decimal_value_correct_after_gc_pressure (1K
    decimal_add+mul + decimal_eq on live Decimal)

Release docs:
- CHANGELOG entry rewritten to cover all three types.
- ROADMAP priority table now down to one entry (#507);
  campaign-progress paragraph notes the campaign is closed.
- KNOWN_ISSUES drops the #575 / #576 placeholder bug rows.
- HISTORY row updated.

Validation: 3,713 pytest, 99 browser parity, mypy clean,
82 conformance, 33 examples, all sync scripts pass.

Co-Authored-By: Claude <noreply@anthropic.invalid>
@aallan aallan changed the title v0.0.134 — Active reclamation of host-store Map handles via heap-wrap-as-ADT (closes #573 phase 1) v0.0.134 — Active reclamation of host-store handles for Map/Set/Decimal (closes #573, #575, #576) May 6, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/test_codegen.py (1)

15685-15699: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

The Decimal rooting helper is observing the wrong signal.

_count_main_pushes() only inspects $main, but the accumulator/element-rooting contract lives in the lowered array_fold_go$... / array_map_go$... helper. On top of that, decimal_count > int_count also counts unrelated wrapper-allocation pushes, so this can stay green even if the actual Decimal accumulator/element shadow-push regresses. Please assert the specific push in the generated helper that carries the Decimal local instead of counting all global.set $gc_sp in $main.

As per coding guidelines, "DO flag: Actual bugs in test logic".

Also applies to: 15718-15745

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_codegen.py` around lines 15685 - 15699, The test currently
inspects only $main for "global.set $gc_sp" which misses the actual
accumulator/element-rooting emitted by the lowered helpers; update the test to
locate the helper function(s) by name pattern (e.g. functions whose names start
with "array_fold_go$" or "array_map_go$") instead of $main, then assert the
helper's gc shadow-push that carries the Decimal local is present (i.e. identify
the specific push site in that helper which stores/advances the gc_sp for the
Decimal local and check for that exact sequence rather than counting all
"global.set $gc_sp" in $main); apply the same fix to the other occurrence noted
(the block around lines 15718-15745).
♻️ Duplicate comments (3)
vera/codegen/api.py (2)

897-933: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Registering the wrapper here is still too early.

Line 917 makes the wrapper visible to Phase 2c before any GC-visible parent points at it. Callers like _alloc_map_wrapper() and the Decimal Option<Decimal> producers allocate the enclosing ADT afterwards, so a GC in that window can evict the brand-new host-store entry and corrupt the returned value. Please root the wrapper across those follow-on allocations, or defer register_wrapper until after the parent object is wired up.


1721-1746: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Gate host_decref_handle off the map-store predicate, not raw Map ops.

Json/Html-only programs now allocate Map wrappers via _alloc_map_wrapper(), so they also need the host_decref_handle import and the kind == 1 cleanup path. As written, _decref_used and the branch on Line 1729 both skip that case, which leaves Json/Html-only modules without the matching decref path.

Suggested fix
-    _decref_used = (
-        result.map_ops_used or result.set_ops_used
-        or result.decimal_ops_used
-    )
+    _map_store_used = (
+        result.map_ops_used
+        or result.json_ops_used
+        or result.html_ops_used
+    )
+    _decref_used = (
+        _map_store_used
+        or result.set_ops_used
+        or result.decimal_ops_used
+    )
     if _decref_used:
         def host_decref_handle(
             _caller: wasmtime.Caller, kind: int, handle: int,
         ) -> None:
-            if kind == 1 and result.map_ops_used:
+            if kind == 1 and _map_store_used:
                 _map_store.pop(handle, None)
             elif kind == 2 and result.set_ops_used:
                 _set_store.pop(handle, None)
             elif kind == 3 and result.decimal_ops_used:
                 _decimal_store.pop(handle, None)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@vera/codegen/api.py` around lines 1721 - 1746, The host_decref_handle import
is currently gated by _decref_used which checks result.map_ops_used, but
Json/Html-only code uses _alloc_map_wrapper and a separate map-store predicate;
update the _decref_used computation to include the map-wrapper store predicate
(the boolean that indicates the presence of the map wrapper store) instead of
relying on result.map_ops_used, and also change the kind == 1 branch inside
host_decref_handle to test that same map-store predicate before popping
_map_store so Json/Html-only modules get the decref path; keep the
linker.define_func call guarded by the revised _decref_used.
vera/browser/runtime.mjs (1)

1196-1219: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Do not register a wrapper before anything roots it.

Line 1208 registers the wrapper immediately, but callers still perform another alloc() before that pointer is reachable from any GC-visible object. If that allocation triggers GC, Phase 2c can decref the fresh handle and leave the enclosing Json/Html/Option object with a dead payload. Please either delay registration until after the parent object stores the pointer, or keep the wrapper temporarily rooted across the remaining allocations.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@vera/browser/runtime.mjs` around lines 1196 - 1219, The current wrapHandle
calls wasm.register_wrapper immediately, which can allow a GC between that
registration and the caller's subsequent alloc/store, leaving the parent object
with a dead payload; change this by removing the immediate registration from
wrapHandle (keep wrapHandle as the simple alloc+writeI32/writeI32) and add a
separate API (e.g. registerWrapper(ptr, kind, rawHandle) or an optional register
flag) so callers like allocMapWrapper / writeJson / writeHtml first create the
wrapper pointer, perform any further allocs/stores that make the wrapper
reachable from the parent, and only then call wasm.register_wrapper via the new
registerWrapper helper; alternatively implement a short-lived root mechanism in
these callers to keep the wrapper pointer rooted across the remaining
allocations, but do not call wasm.register_wrapper until the parent has stored
the pointer.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@CHANGELOG.md`:
- Around line 12-15: Edit the CHANGELOG.md paragraph to narrow the pre-fix
scope: replace the claim that stores "leaked monotonically across all three
types" over repeated `execute()` calls with a statement that the leak occurred
within a single `execute()` lifetime and only affected long-lived execution
contexts/persistent runtimes (e.g., server processes), keeping the rest of the
technical details intact (references to `map_new`/`map_insert`/`map_remove`,
`set_new`/`set_add`/`set_remove`, `Decimal`,
`_map_store`/`_set_store`/`_decimal_store`, the 8-byte wrapper ADT, the
wrap-table, `$gc_collect`, and `host_decref_handle` should remain unchanged).

In `@TESTING.md`:
- Line 9: Update the "**Tests**" table row so the total equals the canonical
pytest collection counts: run pytest tests/ --collect-only -q or python
scripts/check_doc_counts.py to get passed/failed/xfail/xpass/skipped counts,
then regenerate the markdown row (the line containing "**Tests** |") so the
total equals sum of those canonical categories; if there are 4 missing items,
explicitly list them as xfail/xpass (or other categories) in the row to make the
arithmetic correct.

In `@vera/codegen/assembly.py`:
- Around line 323-331: The mark-worklist end (wl_end) is being initialised from
gc_heap_start even when the wrap-table exists, which makes the wrap-table region
act as extra worklist capacity and allows overwrites; instead, when
self._needs_wrap_table (wrap_enabled) is true set the worklist limit to
gc_wraptable_base (the address before the wrap-table) rather than gc_heap_start
so wl_end excludes the wrap-table; update the initialisation logic that computes
gc_wraptable_base, gc_heap_start and wl_end (and the analogous code around the
other occurrence at lines ~844-850) to choose gc_wraptable_base as the worklist
end when wrap_enabled, otherwise use gc_heap_start.

In `@vera/wasm/helpers.py`:
- Around line 269-278: Update the docstring for _is_host_handle_type (and adjust
nearby comments) to reflect that types listed in _HOST_HANDLE_TYPES (e.g., Map,
Set, Decimal) are now GC-managed wrapper ADTs: they are Vera-heap pointers to
8-byte wrapper objects that contain the real i32 host handle in field 0 and
therefore MUST be rooted, rather than opaque host-handle indices excluded from
rooting; remove or replace any text claiming these are opaque indices and
explicitly state the wrapper/roots requirement so subsequent GC/rooting logic
isn’t misled.

---

Outside diff comments:
In `@tests/test_codegen.py`:
- Around line 15685-15699: The test currently inspects only $main for
"global.set $gc_sp" which misses the actual accumulator/element-rooting emitted
by the lowered helpers; update the test to locate the helper function(s) by name
pattern (e.g. functions whose names start with "array_fold_go$" or
"array_map_go$") instead of $main, then assert the helper's gc shadow-push that
carries the Decimal local is present (i.e. identify the specific push site in
that helper which stores/advances the gc_sp for the Decimal local and check for
that exact sequence rather than counting all "global.set $gc_sp" in $main);
apply the same fix to the other occurrence noted (the block around lines
15718-15745).

---

Duplicate comments:
In `@vera/browser/runtime.mjs`:
- Around line 1196-1219: The current wrapHandle calls wasm.register_wrapper
immediately, which can allow a GC between that registration and the caller's
subsequent alloc/store, leaving the parent object with a dead payload; change
this by removing the immediate registration from wrapHandle (keep wrapHandle as
the simple alloc+writeI32/writeI32) and add a separate API (e.g.
registerWrapper(ptr, kind, rawHandle) or an optional register flag) so callers
like allocMapWrapper / writeJson / writeHtml first create the wrapper pointer,
perform any further allocs/stores that make the wrapper reachable from the
parent, and only then call wasm.register_wrapper via the new registerWrapper
helper; alternatively implement a short-lived root mechanism in these callers to
keep the wrapper pointer rooted across the remaining allocations, but do not
call wasm.register_wrapper until the parent has stored the pointer.

In `@vera/codegen/api.py`:
- Around line 1721-1746: The host_decref_handle import is currently gated by
_decref_used which checks result.map_ops_used, but Json/Html-only code uses
_alloc_map_wrapper and a separate map-store predicate; update the _decref_used
computation to include the map-wrapper store predicate (the boolean that
indicates the presence of the map wrapper store) instead of relying on
result.map_ops_used, and also change the kind == 1 branch inside
host_decref_handle to test that same map-store predicate before popping
_map_store so Json/Html-only modules get the decref path; keep the
linker.define_func call guarded by the revised _decref_used.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 911f8549-5ebc-4389-b5b7-501b09b21494

📥 Commits

Reviewing files that changed from the base of the PR and between 69face3 and 957ae95.

📒 Files selected for processing (12)
  • CHANGELOG.md
  • HISTORY.md
  • KNOWN_ISSUES.md
  • README.md
  • ROADMAP.md
  • TESTING.md
  • tests/test_codegen.py
  • vera/browser/runtime.mjs
  • vera/codegen/api.py
  • vera/codegen/assembly.py
  • vera/wasm/calls_containers.py
  • vera/wasm/helpers.py
💤 Files with no reviewable changes (1)
  • KNOWN_ISSUES.md

Comment thread CHANGELOG.md Outdated
Comment thread TESTING.md Outdated
Comment thread vera/codegen/assembly.py
Comment thread vera/codegen/assembly.py
Comment thread vera/wasm/helpers.py
Five findings; verified each.  Skipped 1 + 2 (false claims about
test count and tag count); fixed 3, 4, 5.

Finding 1 (HISTORY.md test count) — SKIPPED.  Reviewer claimed
``pytest --collect-only`` shows 1,608 tests with 15 collection
errors.  Actual: 3,714 tests, 0 errors.  No change needed.

Finding 2 (134 tagged releases / "current git tag count is 0")
— SKIPPED.  ``git tag | wc -l`` shows 134 tags locally (v0.0.1
through v0.0.133, plus the v0.0.134 line is the version in the
current PR which gets tagged after merge per the established
tag-on-branch + CHANGELOG-during-PR pattern).  Reviewer's "0"
claim is wrong.

Finding 3 (test_codegen.py:15886 — Map reclamation bound) —
FIXED.  Tightened the assertion bound from <=100 to <=10 in
``test_map_chain_reclaims_transients``.  Empirically the 10K-
iter chain ends with size=1 (only the live final Map); the
single-digit bound enforces "only the live final Map" semantics
rather than the slacker "GC fires occasionally" reading.

Finding 4 (test_codegen.py:15535 — push regex doesn't verify
sp advance) — FIXED.  Extended the regex in all three positive
shadow-push assertions (Map, Set, Decimal) to require BOTH
halves of the gc_shadow_push idiom in order:
  global.get $gc_sp; local.get N; i32.store      (push)
  global.get $gc_sp; i32.const 4; i32.add;
  global.set $gc_sp                              (advance)
Without the advance, every subsequent push would overwrite the
same slot.  The original ``_assert_param0_not_shadow_pushed``
(negative assertion) is unchanged — if the push isn't there,
no advance either.

Finding 5 (api.py — host_decref_handle gating) — FIXED.  Real
bug: JSON-only and HTML-only programs that didn't directly use
any user-level map_*/set_*/decimal_* op weren't flipping
``_needs_wrap_table`` on the WAT side, so:
  * ``register_wrapper`` wasn't exported
  * ``host_decref_handle`` import wasn't declared
  * Phase 2c of ``$gc_collect`` wasn't emitted

But ``write_json``'s JObject branch and ``write_html``'s
HtmlElement attrs branch internally call ``_alloc_map_wrapper``,
which calls ``$register_wrapper`` via the export.  Pre-fix the
host-side call would silently no-op (the export missing means
``caller["register_wrapper"]`` returns None and we early-return),
leaving the host-store entries to leak indefinitely.

Two-line fix:
* ``vera/codegen/assembly.py`` — flip ``_needs_wrap_table`` if
  ``_json_ops_used or _html_ops_used`` is non-empty.
* ``vera/codegen/api.py`` — extend the ``_decref_used``
  predicate to include ``json_ops_used or html_ops_used`` so
  the destructor host import is defined whenever the WAT side
  references it.

Plus a new ``test_json_only_module_includes_wrap_table``
regression test that asserts a JSON-only program emits the
wrap-table imports, exports, and sweep machinery.

Validation: 3,714 pytest, 99 browser, mypy clean, 82
conformance, 33 examples, all sync scripts pass.

Co-Authored-By: Claude <noreply@anthropic.invalid>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
vera/codegen/api.py (1)

3114-3120: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

host_store_sizes is missing from early IO.exit return paths.

Lines 3114 and 3127 return ExecuteResult without host_store_sizes, so the new introspection field is populated only on normal completion.

Suggested fix
         return ExecuteResult(
             value=None,
             stdout=output_buf.getvalue(),
             stderr=stderr_buf.getvalue() if stderr_buf is not None else "",
             state={k: v[-1] for k, v in state_store.items()},
             exit_code=exit_exc.code,
+            host_store_sizes={k: len(v) for k, v in _host_store_refs.items()},
         )
@@
                 return ExecuteResult(
                     value=None,
                     stdout=output_buf.getvalue(),
                     stderr=stderr_buf.getvalue() if stderr_buf is not None else "",
                     state={k: v[-1] for k, v in state_store.items()},
                     exit_code=cause.code,
+                    host_store_sizes={k: len(v) for k, v in _host_store_refs.items()},
                 )

Also applies to: 3127-3133

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@vera/codegen/api.py` around lines 3114 - 3120, The early return ExecuteResult
in the IO.exit error paths omits the new introspection field host_store_sizes;
update the two ExecuteResult return sites (the one returning value=None with
stdout/stderr/state/exit_code and the other similar return later) to include
host_store_sizes using the same source as the normal completion path (i.e., the
host_store_sizes variable or helper used elsewhere in this module) so the field
is always populated on exit.
♻️ Duplicate comments (2)
vera/codegen/assembly.py (2)

340-346: ⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Stop the mark worklist at the wrap-table boundary.

When wrapper tracking is enabled, $wl_end still comes from $gc_heap_start, so the mark stack can grow through the 64 KiB wrap-table and overwrite registrations before Phase 2c reads them. Use $gc_wrap_base as the worklist limit in wrap-table builds, and keep $gc_heap_start as the heap floor only.

Suggested patch
         wrap_enabled = self._needs_wrap_table
+        wl_end_get = (
+            "    global.get $gc_wrap_base\n"
+            if wrap_enabled
+            else "    global.get $gc_heap_start\n"
+        )
         wrap_locals = (
             "    (local $wrap_read i32)\n"
             "    (local $wrap_write i32)\n"
             "    (local $wrap_kind i32)\n"
             "    (local $wrap_handle i32)\n"
@@
-        # Worklist region sits right after shadow stack, sized to match
+        # Worklist region sits right after the shadow stack and must stop
+        # before the wrap table when wrapper tracking is enabled.
         return (
@@
-            "    global.get $gc_heap_start\n"
+            + wl_end_get
             "    local.set $wl_end\n"

Also applies to: 859-865

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@vera/codegen/assembly.py` around lines 340 - 346, The mark worklist limit is
still using gc_heap_start so the mark stack can grow into the wrap-table; change
the worklist end to stop at the wrap-table boundary by using gc_wraptable_base
(gc_wrap_base/$gc_wrap_base) as the $wl_end when wrapper tracking/wrap-table
builds are enabled, while leaving gc_heap_start as the heap floor; update the
code that computes/assigns wl_end (and any other places setting $wl_end) to
choose gc_wraptable_base when wraptable_overhead is present (also mirror this
fix in the other occurrence referenced in the comment).

726-747: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Phase 2c still depends on a broken liveness signal for wrappers.

This new wrapper sweep assumes Phase 2b’s mark bits are trustworthy, but wrapper ADTs are still scanned word-by-word like ordinary heap objects. Once host-store indices grow into the heap address range, the raw handle slot can satisfy the pointer heuristic and spuriously mark unrelated objects — including dead wrappers — so Phase 2c will retain entries that should have been decref’d. Wrapper objects need a non-scannable mark path here, or the stored handle needs encoding so it cannot alias a heap pointer.

Also applies to: 826-832, 1112-1116

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_codegen.py`:
- Around line 15931-15933: Update the docstring on class
TestHostHandleReclamation573 to accurately reflect that it now contains Map,
Set, and Decimal reclamation regressions (remove "Map only" and the note about
Set/Decimal landing in follow-ups); open the class TestHostHandleReclamation573
and replace the outdated docstring sentence with a concise description that
lists Map, Set, and Decimal reclamation regressions so the class documentation
matches its test contents.
- Around line 16220-16275: Add a sibling test to tests/test_codegen.py mirroring
test_json_only_module_includes_wrap_table but exercising HTML-only ops: create a
small source that calls html_parse (or html_query) only and compiles/runs with
_compile_ok/_run, then assert the generated wat contains the same wrap-table
machinery (import "vera" "host_decref_handle", presence of "$register_wrapper",
and '(export "register_wrapper"') and that the program returns the expected
value; this ensures the _needs_wrap_table flip for html_ops_used is covered and
ties to helpers like _alloc_map_wrapper and _json_ops_used/_html_ops_used.

In `@vera/codegen/api.py`:
- Around line 1729-1734: The host_decref_handle callback incorrectly skips
removing map entries when result.map_ops_used is false, causing leaks for
JSON/HTML-only modules; update host_decref_handle to always pop from _map_store
when kind == 1 regardless of result.map_ops_used (similar to how set handles are
treated for kind == 2), while preserving the existing conditional behavior for
set_ops_used; locate the function host_decref_handle and remove the map_ops_used
gating so _map_store.pop(handle, None) is executed unconditionally for kind ==
1.

---

Outside diff comments:
In `@vera/codegen/api.py`:
- Around line 3114-3120: The early return ExecuteResult in the IO.exit error
paths omits the new introspection field host_store_sizes; update the two
ExecuteResult return sites (the one returning value=None with
stdout/stderr/state/exit_code and the other similar return later) to include
host_store_sizes using the same source as the normal completion path (i.e., the
host_store_sizes variable or helper used elsewhere in this module) so the field
is always populated on exit.

---

Duplicate comments:
In `@vera/codegen/assembly.py`:
- Around line 340-346: The mark worklist limit is still using gc_heap_start so
the mark stack can grow into the wrap-table; change the worklist end to stop at
the wrap-table boundary by using gc_wraptable_base (gc_wrap_base/$gc_wrap_base)
as the $wl_end when wrapper tracking/wrap-table builds are enabled, while
leaving gc_heap_start as the heap floor; update the code that computes/assigns
wl_end (and any other places setting $wl_end) to choose gc_wraptable_base when
wraptable_overhead is present (also mirror this fix in the other occurrence
referenced in the comment).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 30c93878-85bd-4215-a16e-8254f53de1cb

📥 Commits

Reviewing files that changed from the base of the PR and between 957ae95 and 80f6e19.

📒 Files selected for processing (7)
  • HISTORY.md
  • README.md
  • ROADMAP.md
  • TESTING.md
  • tests/test_codegen.py
  • vera/codegen/api.py
  • vera/codegen/assembly.py

Comment thread tests/test_codegen.py Outdated
Comment thread tests/test_codegen.py
Comment thread vera/codegen/api.py
aallan and others added 2 commits May 6, 2026 09:30
Five additional findings beyond round 1's batch.  Verified each;
fixed the critical worklist bug and three smaller items, deferred
one (latent conservative-scan retention) to a follow-up issue.

Finding 8 (assembly.py:346 — CRITICAL: worklist overlapping
wrap-table) — FIXED.  The mark phase used $gc_heap_start as the
worklist upper bound, on the (correct-at-the-time) assumption
that the worklist sat directly before the heap.  After #573
inserted the 64 KiB wrap-table region between worklist and
heap, that assumption broke: the worklist would think it had
``worklist_size + wraptable_size`` of capacity and could grow
into the wrap-table, corrupting (obj_ptr, kind, handle) entries.
Phase 2c would then operate on garbage and evict arbitrary
host-store entries.

Two-line fix:
* New $gc_worklist_end global, always = $gc_stack_limit +
  worklist_size regardless of wrap-table presence.
* $gc_collect's "wl_end" load reads from $gc_worklist_end
  rather than $gc_heap_start.

Finding 6 (CHANGELOG.md — leak-scope wording) — FIXED.  Pre-fix
text read "leaked monotonically across repeated execute()
calls" but each store is execute()-local — the leak is *within*
a single execute(), not across them.  Reworded to clarify that
the failure mode is one long-running execute() (server request
loop, interactive session, Game-of-Life-style multi-generation
program), not a multi-execute() pattern.

Finding 7 (TESTING.md arithmetic) — FIXED.  Row said
"3,714 across 29 files (3,695 passed, 14 skipped)" — the
pass/skip don't add up to 3,714.  Updated to "3,700 passed,
14 skipped" (matches the live pytest output).

Finding 10 (helpers.py docstring) — FIXED.  ``_HOST_HANDLE_TYPES``
is now empty post-#573 phases 1-3, but ``_is_host_handle_type``'s
docstring still described Map / Set / Decimal as "i32 indices
into Python-side host stores".  Rewrote to explain the
historical (pre-#573) purpose AND the post-#573 always-False
return, and why the function is kept rather than deleted (a
hook for future host-handle types added without wrapper
migration).

Finding 9 (assembly.py:832 — wrapper payloads conservatively
scanned) — DEFERRED to follow-up #578.  Phase 2b's word-by-word
scan treats every i32 in a wrapper's payload as a candidate
heap pointer.  For typical programs the host-handle field
(small integer) is well below ``gc_heap_start`` (~147 KiB) so
the heap-range check rejects it, producing no false marks.
For very-long-running programs that allocate hundreds of
thousands of host handles, a handle could exceed the heap-base
threshold and (with the right alignment) be falsely classified
as a heap pointer, spuriously retaining an unrelated heap
object.  Retention bug, not correctness — the falsely-marked
object is still valid; it just sticks around longer than it
should.  Filed as #578 with four candidate fix designs.

Skipped (false claims, addressed in round 1 commit):
* Finding 1 — pytest collects 3,714 tests, not 1,608.
* Finding 2 — git tag count is 134, not 0.

Validation: 3,714 pytest, 99 browser, mypy clean, 82 conformance,
33 examples, all sync scripts pass.

Co-Authored-By: Claude <noreply@anthropic.invalid>
Missed in d33a778 — per the project rule that PR-generated new
issues must land in ROADMAP or KNOWN_ISSUES in the same PR.
#578 was filed as a follow-up for the wrapper-payload scan
issue (round 2 finding 9, deferred for a focused fix); adding
it to the KNOWN_ISSUES bugs table now so it's tracked alongside
#549 and #568.

Co-Authored-By: Claude <noreply@anthropic.invalid>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@vera/codegen/assembly.py`:
- Around line 699-719: The current $register_wrapper function unconditionally
traps when global $gc_wrap_ptr >= global $gc_wrap_end; instead implement a slow
path that, on first fullness, attempts to reclaim dead entries before trapping:
when the overflow check on gc_wrap_ptr vs gc_wrap_end fails, call the existing
compaction/collection routine used in Phase 2c (e.g. invoke $gc_collect or a
dedicated $gc_compact_wraptable helper) while ensuring the new wrapper is
already rooted (store the $ptr/$handle in a temporary root slot or a stack root
so the GC won't collect it), then re-check the gc_wrap_ptr vs gc_wrap_end; only
if it is still full emit the unreachable trap. Update $register_wrapper to
reference gc_wrap_ptr, gc_wrap_end, and call the same Phase 2c compaction path
used by $gc_collect to reclaim entries.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 953d7a07-2dd2-4990-9c4d-e5d8bab2980f

📥 Commits

Reviewing files that changed from the base of the PR and between 80f6e19 and d33a778.

📒 Files selected for processing (4)
  • CHANGELOG.md
  • TESTING.md
  • vera/codegen/assembly.py
  • vera/wasm/helpers.py

Comment thread vera/codegen/assembly.py Outdated
… 1 duplicate

Six findings; verified each.  Fixed 4, skipped the assembly.py:340
finding as duplicate (already addressed in d33a778's
$gc_worklist_end refactor).

Finding A (test_codegen.py:15931 — stale class docstring) —
FIXED.  ``TestHostHandleReclamation573``'s docstring still said
"Map only" with Set/Decimal "in their own follow-ups", but the
class now contains all three.  Rewrote to enumerate the three
failure modes per type (chain reclaims transients, value
correct after pressure, JSON-only / HTML-only wrap-table
inclusion).

Finding B (test_codegen.py:16220 — missing HTML-only test) —
FIXED.  Added ``test_html_only_module_includes_wrap_table`` as
a sibling of ``test_json_only_module_includes_wrap_table``.
Asserts ``host_decref_handle`` import, ``$register_wrapper``
helper, and the export are all present in WAT for an
html_parse-using program.  Note: html_parse transitively pulls
in the prelude's ``html_attr`` (which calls map_get) so
``_map_ops_used`` ends up set anyway in practice — but the
``_needs_wrap_table`` flip on ``_html_ops_used`` is the load-
bearing fix if that prelude transitivity ever changes (e.g.
if a future codegen DCE eliminates unused prelude functions).

Finding C (api.py:1729 — host_decref_handle map_ops_used gating)
— FIXED.  Real bug: ``kind == 1 and result.map_ops_used`` skips
the pop for JSON-only / HTML-only programs (where
``map_ops_used`` is False but JObject / HtmlElement allocations
still create Map wrappers — ``_map_store`` exists because
its creation is gated on ``map_ops_used or json_ops_used or
html_ops_used``).  Drop the ``map_ops_used`` part of the gate;
the runtime invariant "kind == 1 ⟹ _map_store exists" holds
because the wrap-table only emits kind=1 when the Map store
is created.  ``kind == 2`` and ``kind == 3`` keep their gates
because their stores have a single creation condition.

Finding D (api.py:3114 outside-of-diff — ExecuteResult missing
host_store_sizes in IO.exit early returns) — FIXED.  The two
early-return ``ExecuteResult(...)`` sites in the IO.exit error
paths omitted ``host_store_sizes``, leaving the field empty
even when host stores existed.  Threaded
``{k: len(v) for k, v in _host_store_refs.items()}`` through
both sites so the field is always populated, mirroring the
normal-completion path.

Skipped:
* assembly.py:340 (duplicate) — reviewer suggests using
  ``gc_wraptable_base`` as the worklist end.  This is
  mathematically equivalent to the ``$gc_worklist_end`` global
  introduced in d33a778's round-2 fix
  (``gc_worklist_end = gc_stack_limit + worklist_size`` and
  ``gc_wraptable_base = data_end + gc_stack_size +
  worklist_size``; they refer to the same address when wrap-
  table is enabled and the new constant is independent of
  whether wrap-table is enabled).  Already fixed.

Validation: 3,715 pytest, 99 browser, mypy clean, 82
conformance, 33 examples, all sync scripts pass.

Co-Authored-By: Claude <noreply@anthropic.invalid>
CodeRabbit round 4 finding: $register_wrapper traps on overflow
instead of trying $gc_collect compaction first.  Real
limitation but bounded in practice (every wrapper is also a
heap allocation so wrap-table-full and heap-full happen at
similar cadences and GC fires first; 10K-iter Map stress test
validates).  Implementation requires re-entrancy-safe GC from
inside register_wrapper which warrants its own design + new
stress tests.

Filed as #579 with full design sketch.  Added to KNOWN_ISSUES
per the project rule that PR-generated new issues must land in
ROADMAP or KNOWN_ISSUES in the same PR.

Co-Authored-By: Claude <noreply@anthropic.invalid>
@aallan

aallan commented May 6, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Finding verified — $register_wrapper does trap on overflow without trying compaction. Real limitation but deferred to follow-up #579 in commit 6e2e9ce.

Why deferred:

  • Bounded by heap fill rate in practice — every wrapper is also a heap allocation (8-byte body + 4-byte header ≈ 16 bytes). 4 096 wrappers ≈ 64 KiB heap consumed, which is one page. Heap fills around the same time the wrap-table fills, so $alloc triggers $gc_collect first. The 10K-iter array_fold stress tests in TestHostHandleReclamation573 validate this empirically (10× the wrap-table capacity, no trap, post-GC residual ≤ 10 for Map / < 2K for Set / < 1.5K for Decimal).
  • Implementation requires re-entrancy-safe GC from inside $register_wrapper, with the new wrapper rooted via shadow-stack push before $gc_collect runs. Non-trivial design — adding a full GC prologue/epilogue to a hot-path function — that warrants its own PR with new stress tests covering the slow path.

What #579 covers:

  • Two-step register design sketch (push temp root → $gc_collect → pop temp root → re-check → trap if still full)
  • Three subtleties to design (re-entrancy under arbitrary caller shadow-stack state, hot-path cost, test coverage)
  • Acceptance criteria including a wrapper-heavy synthetic stress test

Added to KNOWN_ISSUES.md per the project rule that PR-generated new issues land in ROADMAP / KNOWN_ISSUES in the same PR.

@coderabbitai

coderabbitai Bot commented May 6, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

@aallan Thanks for the thorough write-up. The deferral rationale is sound:

  • The heap-fill-before-wrap-table-fill invariant (≈16 bytes/wrapper means ~64 KiB heap consumed before the 4096-entry table fills) is the key correctness argument for the current PR, and the 10K-iter stress tests in TestHostHandleReclamation573 provide solid empirical backing.
  • The re-entrancy constraint is real — pushing a temp root, triggering $gc_collect from inside $register_wrapper, popping it, and then re-checking under arbitrary caller shadow-stack state is genuinely non-trivial, and conflating that design into this PR would have blurred the scope.

The two-step sketch in #579 (push temp root → $gc_collect → pop → re-check → trap if still full) is the right structure. One thing worth pinning in the acceptance criteria for #579: the stress test should exercise the slow path explicitly (i.e., drive wrap-table saturation before heap exhaustion triggers $gc_collect via $alloc), since the current empirical safety net only validates the common-case ordering.


✏️ Learnings added
Learnt from: aallan
URL: https://github.com/aallan/vera/pull/577

Timestamp: 2026-05-06T08:56:03.879Z
Learning: In vera/codegen/assembly.py, `$register_wrapper` traps on wrap-table overflow (4096-entry, 64 KiB region) without attempting in-place compaction. This is intentional and safe in practice because every wrapper is also an 8-byte heap allocation (≈16 bytes total), so the heap fills and `$alloc` triggers `$gc_collect` before the wrap-table saturates. Re-entrancy-safe GC inside `$register_wrapper` (requiring a shadow-stack temp root before `$gc_collect` runs) is deferred to follow-up issue `#579`. Documented in KNOWN_ISSUES.md.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

Round-4 finding from CodeRabbit, originally deferred in 6e2e9ce
but the user pushed back asking whether to defer or fix.  The
implementation is ~20 lines of WAT and the re-entrancy concerns
are tractable, so fix it in this PR rather than carrying #579
as a known issue.

Pre-fix $register_wrapper traps with `unreachable` the moment
$gc_wrap_ptr >= $gc_wrap_end — even if Phase 2c compaction
would have freed thousands of dead entries.

Post-fix slow path on overflow:
1. Push the in-flight wrapper ptr onto the shadow stack as a
   temporary root (with overflow check matching gc_shadow_push).
   Without this, Phase 2b would mark the in-flight wrapper
   unreachable and Phase 3 would link it to the free list — we'd
   then append to a freed object after the collect.
2. Call $gc_collect (Phase 2c walks the wrap-table, drops dead
   entries, resets $gc_wrap_ptr to the compacted end).
3. Pop the temporary root.
4. Re-check the overflow condition; trap only if still full
   (which now indicates 4096+ genuinely-live wrappers and a
   real out-of-budget condition).

Re-entrancy: $gc_collect's Phase 2c calls
$vera.host_decref_handle (host import) for unmarked wrappers
but doesn't allocate or wrap, so no recursion through
$register_wrapper.

Test coverage: structural pin
`test_register_wrapper_has_compaction_slow_path` asserts the
slow path is wired up — verifies the $gc_collect call,
shadow-push of $ptr before the call, and the post-compaction
re-check are all present in the generated WAT.  End-to-end
exercise of the slow path is hard (every wrapper is also a
heap allocation, so wrap-table-full and heap-full happen at
similar cadences and $alloc triggers GC first under normal
conditions); the structural test catches refactors that revert
to the unconditional trap.

Closes #579.  Removed from KNOWN_ISSUES.  CHANGELOG entry
extended to cover the slow-path fix in addition to the original
phase 1-3 migration.

Validation: 3,716 pytest, mypy clean, doc counts consistent.

Co-Authored-By: Claude <noreply@anthropic.invalid>
@aallan

aallan commented May 6, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Reversal — actually fixed in 2ef0e60 (was deferred in 6e2e9ce). The user pushed back on the defer, the implementation is ~20 lines of WAT, and the re-entrancy concerns are tractable, so fixed in this PR.

Slow path:

  1. On overflow, shadow-push the in-flight wrapper as a temporary root. Without this, Phase 2b marks it unreachable and Phase 3 frees it before we append.
  2. Call $gc_collect. Phase 2c walks the wrap-table, drops dead entries, resets $gc_wrap_ptr to the compacted end.
  3. Pop the temporary root.
  4. Re-check overflow; trap only if still full (which now genuinely means 4 096+ live wrappers).

Re-entrancy verified: $gc_collect's Phase 2c calls $vera.host_decref_handle (a host import) for unmarked wrappers but doesn't allocate or wrap, so no recursion through $register_wrapper.

Test: new test_register_wrapper_has_compaction_slow_path structural pin — asserts the WAT contains the $gc_collect call, the shadow-push of $ptr before the call, and the post-compaction re-check. End-to-end exercise is hard (heap-fill and wrap-table-fill happen at similar rates so $alloc triggers GC first under normal conditions); the structural test catches refactors reverting to the unconditional trap.

#579 now in the PR's auto-close footer alongside #573/#575/#576. Removed from KNOWN_ISSUES; CHANGELOG entry extended.

@coderabbitai

coderabbitai Bot commented May 6, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

@aallan Excellent — the implementation is clean and the correctness argument holds up well:

  • Shadow-push before $gc_collect is the essential invariant. The in-flight $ptr is not yet in the wrap-table, so Phase 2b would mark it unreachable and Phase 3 would reclaim it without the temp root. The ordering (push → collect → pop → re-check) is correct.
  • Re-entrancy argument is tight. Phase 2c only calls host_decref_handle to evict store entries — it neither allocates heap cells nor calls $register_wrapper — so the call graph is a DAG and there's no recursive re-entry path.
  • Trap-if-still-full post-compaction is the right termination condition: at that point all 4096+ table entries correspond to live wrappers (survived Phase 2b marking), so GC genuinely cannot help and a trap is the correct response.
  • Structural test rationale is sound. Given that $alloc empirically triggers $gc_collect before wrap-table saturation under normal heap pressure, a WAT-level structural pin ($gc_collect call + shadow-push of $ptr + post-compaction re-check present) is the right instrument to guard against future regressions to the unconditional-trap form.

One thing to double-check (if not already covered): the pop of the temp root happens unconditionally — i.e., even on the trap branch — so the shadow-stack is never left in a dirty state on abnormal exit. Typically in WAT this would be handled via a block/br that pops before the unreachable, or by placing the pop before the re-check guard. Worth confirming the shadow-stack is balanced on the trap path.

Updating the learning to reflect the fix.


✏️ Learnings added
Learnt from: aallan
URL: https://github.com/aallan/vera/pull/577

Timestamp: 2026-05-06T09:06:56.039Z
Learning: In vera/codegen/assembly.py, `$register_wrapper` now has a compaction slow path (added in commit `2ef0e60` of PR `#577`) instead of unconditionally trapping on wrap-table overflow. On overflow: (1) shadow-push the in-flight wrapper as a temporary GC root (critical — it is not yet in the wrap-table so Phase 2b would otherwise mark it unreachable), (2) call `$gc_collect` (Phase 2c walks the wrap-table, drops unmarked entries, resets `$gc_wrap_ptr`), (3) pop the temp root, (4) re-check; trap only if still full (meaning 4096+ genuinely live wrappers). Re-entrancy is safe because Phase 2c only calls `host_decref_handle` (a host import) and never allocates or calls `$register_wrapper`. Guarded by structural test `test_register_wrapper_has_compaction_slow_path`. Issue `#579` is closed by this PR.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

CodeRabbit follow-up on #579's slow-path implementation:
"the pop of the temp root happens unconditionally — i.e., even
on the trap branch — so the shadow-stack is never left in a
dirty state on abnormal exit."

The current implementation already has this property: the pop
runs immediately after $gc_collect returns and BEFORE the
re-check guard, so on trap the pop has already executed.
`unreachable` aborts the WASM module so the imbalance has no
observable effect today, but treating WAT shadow-stack
discipline as a hard invariant catches regressions before any
future change converts the trap to a recoverable error path.

Extended `test_register_wrapper_has_compaction_slow_path` to
match the pop sequence appearing strictly between
`call $gc_collect` and the re-check `i32.ge_u` — a structural
pin that fails if a refactor reorders the slow path's tail.

Co-Authored-By: Claude <noreply@anthropic.invalid>
@aallan

aallan commented May 6, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Verified — the shadow stack is balanced on the trap path. Tracing both trap sites in $register_wrapper:

Trap 1 (shadow-stack-limit overflow check, before any push): if (gc_sp >= gc_stack_limit) { unreachable }. No push has happened yet, so trapping leaves the shadow stack at the same level as caller-entry. Balanced.

Trap 2 (post-compaction re-check): the slow path is laid out as

;; push (root the in-flight wrapper)
global.get $gc_sp; local.get $ptr; i32.store
global.get $gc_sp; i32.const 4; i32.add; global.set $gc_sp

call $gc_collect

;; pop (always runs)
global.get $gc_sp; i32.const 4; i32.sub; global.set $gc_sp

;; re-check; trap if still full
global.get $gc_wrap_ptr; global.get $gc_wrap_end; i32.ge_u
if unreachable end

The pop runs unconditionally between $gc_collect and the re-check guard, so when the second unreachable fires, the pop has already executed — $gc_sp is back to caller-entry level. Balanced.

Today the trap aborts the WASM module so any imbalance is unobservable, but I agree treating shadow-stack discipline as a hard invariant catches regressions if a future change converts the trap to a recoverable error path. Pinned with a structural test in 20d4f77test_register_wrapper_has_compaction_slow_path now asserts the pop sequence appears strictly between call $gc_collect and the re-check i32.ge_u, so any reordering of the slow path's tail (e.g. moving the pop after the re-check) fails the test.

@coderabbitai

coderabbitai Bot commented May 6, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

@aallan Verification is thorough and the layout is correct on both paths.

  • Trap 1 — pre-push guard: trivially balanced; nothing to unwind.
  • Trap 2 — post-compaction guard: the unconditional pop between call $gc_collect and the re-check i32.ge_u is the key structural fact. Since the pop is before the conditional if unreachable end, both the fall-through path (table has space, function returns normally) and the trap path execute the pop exactly once. $gc_sp is at caller-entry level in both cases.

The ordering pin in test_register_wrapper_has_compaction_slow_path (20d4f77) is exactly the right instrument — asserting the pop appears strictly between call $gc_collect and the i32.ge_u catches the one refactor that would break the invariant (moving the pop after the guard), which raw presence-of-pop checks would miss. Good call pinning the ordering rather than just existence.


✏️ Learnings added
Learnt from: aallan
URL: https://github.com/aallan/vera/pull/577

Timestamp: 2026-05-06T09:22:38.248Z
Learning: In vera/codegen/assembly.py, `$register_wrapper`'s slow-path shadow-stack discipline is verified balanced on both trap sites (PR `#577`, commit `20d4f77`): Trap 1 (shadow-stack-limit guard) fires before any push — no imbalance. Trap 2 (post-compaction re-check) has the pop (`gc_sp -= 4`) unconditionally between `call $gc_collect` and the `i32.ge_u if unreachable end` guard, so `$gc_sp` is at caller-entry level when `unreachable` fires. Structural test `test_register_wrapper_has_compaction_slow_path` asserts the pop sequence appears strictly between `call $gc_collect` and the re-check `i32.ge_u`, catching any reordering that would break the invariant if the trap is ever converted to a recoverable error path.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

@aallan aallan merged commit 571600c into main May 6, 2026
20 checks passed
@aallan aallan deleted the claude/fix-573-handle-reclamation branch May 6, 2026 09:34
aallan added a commit that referenced this pull request May 9, 2026
… 1 duplicate

Six findings; verified each.  Fixed 4, skipped the assembly.py:340
finding as duplicate (already addressed in d33a778's
$gc_worklist_end refactor).

Finding A (test_codegen.py:15931 — stale class docstring) —
FIXED.  ``TestHostHandleReclamation573``'s docstring still said
"Map only" with Set/Decimal "in their own follow-ups", but the
class now contains all three.  Rewrote to enumerate the three
failure modes per type (chain reclaims transients, value
correct after pressure, JSON-only / HTML-only wrap-table
inclusion).

Finding B (test_codegen.py:16220 — missing HTML-only test) —
FIXED.  Added ``test_html_only_module_includes_wrap_table`` as
a sibling of ``test_json_only_module_includes_wrap_table``.
Asserts ``host_decref_handle`` import, ``$register_wrapper``
helper, and the export are all present in WAT for an
html_parse-using program.  Note: html_parse transitively pulls
in the prelude's ``html_attr`` (which calls map_get) so
``_map_ops_used`` ends up set anyway in practice — but the
``_needs_wrap_table`` flip on ``_html_ops_used`` is the load-
bearing fix if that prelude transitivity ever changes (e.g.
if a future codegen DCE eliminates unused prelude functions).

Finding C (api.py:1729 — host_decref_handle map_ops_used gating)
— FIXED.  Real bug: ``kind == 1 and result.map_ops_used`` skips
the pop for JSON-only / HTML-only programs (where
``map_ops_used`` is False but JObject / HtmlElement allocations
still create Map wrappers — ``_map_store`` exists because
its creation is gated on ``map_ops_used or json_ops_used or
html_ops_used``).  Drop the ``map_ops_used`` part of the gate;
the runtime invariant "kind == 1 ⟹ _map_store exists" holds
because the wrap-table only emits kind=1 when the Map store
is created.  ``kind == 2`` and ``kind == 3`` keep their gates
because their stores have a single creation condition.

Finding D (api.py:3114 outside-of-diff — ExecuteResult missing
host_store_sizes in IO.exit early returns) — FIXED.  The two
early-return ``ExecuteResult(...)`` sites in the IO.exit error
paths omitted ``host_store_sizes``, leaving the field empty
even when host stores existed.  Threaded
``{k: len(v) for k, v in _host_store_refs.items()}`` through
both sites so the field is always populated, mirroring the
normal-completion path.

Skipped:
* assembly.py:340 (duplicate) — reviewer suggests using
  ``gc_wraptable_base`` as the worklist end.  This is
  mathematically equivalent to the ``$gc_worklist_end`` global
  introduced in d33a778's round-2 fix
  (``gc_worklist_end = gc_stack_limit + worklist_size`` and
  ``gc_wraptable_base = data_end + gc_stack_size +
  worklist_size``; they refer to the same address when wrap-
  table is enabled and the new constant is independent of
  whether wrap-table is enabled).  Already fixed.

Validation: 3,715 pytest, 99 browser, mypy clean, 82
conformance, 33 examples, all sync scripts pass.

Co-Authored-By: Claude <noreply@anthropic.invalid>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant