…ives

Pins the property whose absence got PR boxlite-ai#520's global waitpid(-1) reaper
reverted (Issue boxlite-ai#523, criterion #2): a child the reaper never registered
must be left for its owner to wait(). Two-side verified — injecting a
global waitpid(-1) into the sweep makes the owner's wait() return ECHILD
("No child processes") and the test fail; the scoped sweep preserves
exit code 42.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds `ShimReaper`, a per-runtime worker that periodically calls
`waitpid(pid, WNOHANG)` on a small registered PID set. Replaces the
reverted daemon-wide reaper attempt from PR boxlite-ai#520 with a scoped design
that matches Issue boxlite-ai#523's acceptance criteria.

Problem
=======

`boxlite-shim` children that get abandoned mid-init (panic past
`CleanupGuard`, retry loop spawning a fresh shim, reattach paths where
the Rust-side `Child` was never wait()'d) accumulate as `<defunct>` PIDs
under the daemon. The `CL84LvGx7RBE` incident on dev showed 7+ zombies
piled up over time, and the contributing cause sat on top of the
CleanupGuard fix (which itself is independent and stays).

PR boxlite-ai#520 originally landed a `waitpid(-1, WNOHANG)` reaper to drain these.
That call races every other `Child::wait()` site in the workspace: if
the reaper wins, the owner's `wait()` returns ECHILD and the exit code
is lost. The reaper was reverted on the same PR pending the scoped
design this commit implements.

Design
======

`src/boxlite/src/util/reaper.rs` introduces:

- `ShimReaper` — owns a `std::thread` worker that scans an `Arc<Mutex<HashSet<u32>>>`
  registry every 250 ms and reaps any registered PID whose `waitpid`
  reports exit or ECHILD. Std thread (not tokio task) so it works
  regardless of whether `RuntimeImpl::new` was called inside a tokio
  runtime context.
- `ReaperHandle` — RAII registration handle. Drop removes the PID from
  the registry; a panic in any caller path that owns one can't leak.
- `shutdown()` — Condvar-driven, completes in well under one tick so
  `RuntimeImpl::shutdown` (sync and async paths both call it) doesn't
  block the runtime exit on a stuck reaper.

Why scoped, not daemon-wide
---------------------------

Per Issue boxlite-ai#523, an unscoped reaper races every `Child::wait()` call site.
This reaper only touches PIDs that were explicitly registered. Today's
only registrar is `ShimHandler::from_spawned`. The `let _ = process.wait();`
sites in `shim.rs:112` / `shim.rs:121` discard their results, so
ECHILD-from-reaper-win is safe. Audit of every `Child::wait()` /
`Child::try_wait()` call site in `src/boxlite/`:

| Site                                | Process    | Race-safe?       |
|-------------------------------------|------------|------------------|
| shim.rs:102 (stop's try_wait loop)  | shim       | Yes — `Err(_)` arm falls through to kill+wait, both ECHILD-discarded |
| shim.rs:112, 121 (`let _ = wait()`) | shim       | Yes — result discarded |
| watchdog.rs:256                     | test only  | N/A              |
| net/libslirp.rs:135                 | gvproxy    | Not registered, reaper doesn't touch |
| box_impl.rs:1169 (ChildGuard)       | test only  | N/A              |
| ProcessMonitor::try_wait            | any        | Already maps ECHILD to `Unknown` |

Why polling, not SIGCHLD
------------------------

SIGCHLD plumbing in async Rust (signal-hook / tokio::signal::unix) is
process-global and adds a race surface against the runtime's other
signal handlers. A 250 ms HashSet scan is cheaper than that complexity
buys back. Worst-case zombie lifetime is 250 ms; tests verify drain in
< 2 s by polling, never sleeping.

Wiring
======

- `RuntimeImpl` gains `shim_reaper: Arc<ShimReaper>`, spawned in
  `RuntimeImpl::new`, shut down in both `shutdown()` (async path) and
  `shutdown_sync()` (atexit / Drop path).
- `ShimController::new` takes `Arc<ShimReaper>`; threaded through from
  `vmm_spawn::spawn_vm` via `ctx.runtime.shim_reaper`.
- `ShimHandler::from_spawned` registers the PID and stores the
  `ReaperHandle`. `ShimHandler::from_pid` (reattach mode) does NOT
  register — `waitpid` on a non-child returns ECHILD anyway; the
  reattached shim's eventual zombie belongs to init, not us.

Tests
=====

Acceptance criteria from Issue boxlite-ai#523, restated and pinned:

- [x] Scopes to known shim PIDs, never calls `waitpid(-1, ...)` — see
      `probe_pid` in reaper.rs.
- [x] Audit of `Child::wait()` call sites — recorded in the table above.
- [x] Shutdown path tied to `RuntimeImpl::shutdown` — both async and
      sync paths call `shim_reaper.shutdown()`. Drop on the reaper is
      idempotent.
- [x] Verification test runs in < 2 s (poll-based) — three unit tests
      in `reaper.rs::tests` and two integration tests in
      `src/boxlite/tests/zombie_reaper.rs::n_init_failures_leave_zero_zombies`,
      `registry_accepts_re_registration_after_drop`. Wallclock for each
      is sub-second on the dev host.
- [x] Integration test demonstrates "N init failures leave 0 zombies" —
      `n_init_failures_leave_zero_zombies` spawns N=8 (matching the
      `CL84LvGx7RBE` count), forgets each Child to mirror the abandoned-
      mid-init path, and asserts both registry drain and `/proc/<pid>/status`
      absence-or-not-Z.

Test counts:
- Unit (reaper.rs): 3 tests, all pass in ~0.3 s total
- Integration (tests/zombie_reaper.rs): 2 tests, all pass in ~0.5 s total
- Regression: 754/754 boxlite lib tests pass with --features rest
  (excluding the pre-existing `ws_watchdog` flake unrelated to this
  change); 118/118 boxlite-cli unit tests pass.

Out of scope
============

- Reattach orphans: a reattached `ShimHandler` doesn't register with
  the reaper. If the original spawning process is gone, init will reap
  the eventual zombie. Tracking via reconciler (separate epic) covers
  the DB/runtime drift case.
- PR boxlite-ai#573 (orphan shim port retention, Issue boxlite-ai#565): independent fix
  for a different leak mechanism (`is_same_process` recovery
  misidentification). Both are needed for the full dind test reliability
  story; landing one doesn't subsume the other.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ives

Pins the property whose absence got PR boxlite-ai#520's global waitpid(-1) reaper
reverted (Issue boxlite-ai#523, criterion #2): a child the reaper never registered
must be left for its owner to wait(). Two-side verified — injecting a
global waitpid(-1) into the sweep makes the owner's wait() return ECHILD
("No child processes") and the test fail; the scoped sweep preserves
exit code 42.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds `ShimReaper`, a per-runtime worker that periodically calls
`waitpid(pid, WNOHANG)` on a small registered PID set. Replaces the
reverted daemon-wide reaper attempt from PR boxlite-ai#520 with a scoped design
that matches Issue boxlite-ai#523's acceptance criteria.

Problem
=======

`boxlite-shim` children that get abandoned mid-init (panic past
`CleanupGuard`, retry loop spawning a fresh shim, reattach paths where
the Rust-side `Child` was never wait()'d) accumulate as `<defunct>` PIDs
under the daemon. The `CL84LvGx7RBE` incident on dev showed 7+ zombies
piled up over time, and the contributing cause sat on top of the
CleanupGuard fix (which itself is independent and stays).

PR boxlite-ai#520 originally landed a `waitpid(-1, WNOHANG)` reaper to drain these.
That call races every other `Child::wait()` site in the workspace: if
the reaper wins, the owner's `wait()` returns ECHILD and the exit code
is lost. The reaper was reverted on the same PR pending the scoped
design this commit implements.

Design
======

`src/boxlite/src/util/reaper.rs` introduces:

- `ShimReaper` — owns a `std::thread` worker that scans an `Arc<Mutex<HashSet<u32>>>`
  registry every 250 ms and reaps any registered PID whose `waitpid`
  reports exit or ECHILD. Std thread (not tokio task) so it works
  regardless of whether `RuntimeImpl::new` was called inside a tokio
  runtime context.
- `ReaperHandle` — RAII registration handle. Drop removes the PID from
  the registry; a panic in any caller path that owns one can't leak.
- `shutdown()` — Condvar-driven, completes in well under one tick so
  `RuntimeImpl::shutdown` (sync and async paths both call it) doesn't
  block the runtime exit on a stuck reaper.

Why scoped, not daemon-wide
---------------------------

Per Issue boxlite-ai#523, an unscoped reaper races every `Child::wait()` call site.
This reaper only touches PIDs that were explicitly registered. Today's
only registrar is `ShimHandler::from_spawned`. The `let _ = process.wait();`
sites in `shim.rs:112` / `shim.rs:121` discard their results, so
ECHILD-from-reaper-win is safe. Audit of every `Child::wait()` /
`Child::try_wait()` call site in `src/boxlite/`:

| Site                                | Process    | Race-safe?       |
|-------------------------------------|------------|------------------|
| shim.rs:102 (stop's try_wait loop)  | shim       | Yes — `Err(_)` arm falls through to kill+wait, both ECHILD-discarded |
| shim.rs:112, 121 (`let _ = wait()`) | shim       | Yes — result discarded |
| watchdog.rs:256                     | test only  | N/A              |
| net/libslirp.rs:135                 | gvproxy    | Not registered, reaper doesn't touch |
| box_impl.rs:1169 (ChildGuard)       | test only  | N/A              |
| ProcessMonitor::try_wait            | any        | Already maps ECHILD to `Unknown` |

Why polling, not SIGCHLD
------------------------

SIGCHLD plumbing in async Rust (signal-hook / tokio::signal::unix) is
process-global and adds a race surface against the runtime's other
signal handlers. A 250 ms HashSet scan is cheaper than that complexity
buys back. Worst-case zombie lifetime is 250 ms; tests verify drain in
< 2 s by polling, never sleeping.

Wiring
======

- `RuntimeImpl` gains `shim_reaper: Arc<ShimReaper>`, spawned in
  `RuntimeImpl::new`, shut down in both `shutdown()` (async path) and
  `shutdown_sync()` (atexit / Drop path).
- `ShimController::new` takes `Arc<ShimReaper>`; threaded through from
  `vmm_spawn::spawn_vm` via `ctx.runtime.shim_reaper`.
- `ShimHandler::from_spawned` registers the PID and stores the
  `ReaperHandle`. `ShimHandler::from_pid` (reattach mode) does NOT
  register — `waitpid` on a non-child returns ECHILD anyway; the
  reattached shim's eventual zombie belongs to init, not us.

Tests
=====

Acceptance criteria from Issue boxlite-ai#523, restated and pinned:

- [x] Scopes to known shim PIDs, never calls `waitpid(-1, ...)` — see
      `probe_pid` in reaper.rs.
- [x] Audit of `Child::wait()` call sites — recorded in the table above.
- [x] Shutdown path tied to `RuntimeImpl::shutdown` — both async and
      sync paths call `shim_reaper.shutdown()`. Drop on the reaper is
      idempotent.
- [x] Verification test runs in < 2 s (poll-based) — three unit tests
      in `reaper.rs::tests` and two integration tests in
      `src/boxlite/tests/zombie_reaper.rs::n_init_failures_leave_zero_zombies`,
      `registry_accepts_re_registration_after_drop`. Wallclock for each
      is sub-second on the dev host.
- [x] Integration test demonstrates "N init failures leave 0 zombies" —
      `n_init_failures_leave_zero_zombies` spawns N=8 (matching the
      `CL84LvGx7RBE` count), forgets each Child to mirror the abandoned-
      mid-init path, and asserts both registry drain and `/proc/<pid>/status`
      absence-or-not-Z.

Test counts:
- Unit (reaper.rs): 3 tests, all pass in ~0.3 s total
- Integration (tests/zombie_reaper.rs): 2 tests, all pass in ~0.5 s total
- Regression: 754/754 boxlite lib tests pass with --features rest
  (excluding the pre-existing `ws_watchdog` flake unrelated to this
  change); 118/118 boxlite-cli unit tests pass.

Out of scope
============

- Reattach orphans: a reattached `ShimHandler` doesn't register with
  the reaper. If the original spawning process is gone, init will reap
  the eventual zombie. Tracking via reconciler (separate epic) covers
  the DB/runtime drift case.
- PR boxlite-ai#573 (orphan shim port retention, Issue boxlite-ai#565): independent fix
  for a different leak mechanism (`is_same_process` recovery
  misidentification). Both are needed for the full dind test reliability
  story; landing one doesn't subsume the other.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ives

Pins the property whose absence got PR boxlite-ai#520's global waitpid(-1) reaper
reverted (Issue boxlite-ai#523, criterion #2): a child the reaper never registered
must be left for its owner to wait(). Two-side verified — injecting a
global waitpid(-1) into the sweep makes the owner's wait() return ECHILD
("No child processes") and the test fail; the scoped sweep preserves
exit code 42.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The reaper avoids accidentally consuming a kernel-recycled PID by relying
on one property: once `waitpid(pid, WNOHANG)` succeeds, `pid` is removed
from the registry **before the next sweep tick can fire `waitpid(pid)`
again**. If that eviction is atomic w.r.t. the sweep loop, no future PID
assignment can land back in the registry without an explicit `register()`
call.

Adds `reaped_pid_evicted_atomically_so_kernel_pid_reuse_is_safe`:

Part 1 — spawn + register + wait for `registered()` to evict the PID,
then sleep `3 × REAPER_TICK` and re-assert the registry is still empty
for that PID. Any sweep-vs-eviction race (e.g. retaining the sweep
snapshot across ticks, or re-adding from a stale Vec) would surface as
the PID re-appearing.

Part 2 — spawn an unregistered child with the distinctive exit code 73
and `child.wait()` it across many sweep ticks. The exit code must
survive. This is the production safety property: even on the off chance
the kernel recycles the just-reaped PID to this child, the reaper's
registry doesn't contain it, so the sweep skips it.

Two-side verified on this branch:

| step | code state                                          | result                                                                                          |
|------|------------------------------------------------------|-------------------------------------------------------------------------------------------------|
| A    | scoped `waitpid(pid, …)`                            | `ok` (1.87 s)                                                                                   |
| B    | unconditional `while waitpid(-1, …) > 0 {}` at sweep entry — the PR boxlite-ai#520 reversal | **FAIL** at part 2: `owner of unregistered PID 676048 could not wait(): No child processes (os error 10)` — ECHILD, exit code 73 lost |
| C    | restored                                            | `ok` (1.87 s)                                                                                   |

Why this complements `unregistered_child_exit_code_is_not_stolen`:
- That test injects `waitpid(-1)` directly into `probe_pid` to prove the
  reaper's per-PID `waitpid` is scoped. Sufficient for the "don't broaden
  the waitpid call" angle.
- This test pins the *adjacent* invariant — that registry eviction is
  atomic relative to subsequent sweep ticks — which is what makes
  kernel PID reuse safe even if the `waitpid` call site itself stays
  scoped. The two together cover both ways someone could re-introduce
  the PR boxlite-ai#520 regression: broaden the call, or fail to evict in time.

clippy `-D warnings` clean, fmt:check clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>