Summary

Fix three confirmed hermes gateway run crashes on Windows caused by unreliable os.kill(pid, 0) behavior, plus a latent correctness bug in the browser orphan reaper. Introduce a cross-platform pid_is_alive() helper that routes through the Win32 API on Windows and preserves traditional os.kill semantics on POSIX.

The problem

os.kill(pid, 0) — the canonical POSIX liveness probe — is unreliable on Windows in both directions:

1. False negative (crash). It can raise OSError(WinError 11) ("An attempt was made to load a program with an incorrect format") for perfectly healthy processes when the target's arch differs from the caller, or raise OSError(WinError 87) / SystemError for PIDs the kernel rejects as malformed.
2. False positive (silent). It can return success for processes that have already exited if any handle to them remains open, masking the exit and causing callers to treat dead PIDs as alive.

Reproduction (the crash path)

On a clean Windows 11 + Python 3.11 install, three consecutive hermes gateway run invocations crashed at three different os.kill(pid, 0) sites, each leaving a stale PID/lock file that detonated the next probe:

OSError: [WinError 11] An attempt was made to load a program with an incorrect format
  at gateway/status.py:578 in get_running_pid
  at gateway/status.py:343 in acquire_scoped_lock
OSError: [WinError 87] / SystemError: <built-in function kill> returned a result with an exception set
  at gateway/status.py:343 in acquire_scoped_lock (subsequent run)

Between crashes, stale files have to be manually removed:

rm ~/.hermes/gateway.pid
rm ~/.local/state/hermes/gateway-locks/*.lock

The silent bug

tools/browser_tool.py::_reap_orphaned_browser_sessions uses os.kill(owner_pid, 0) to decide whether a browser daemon's owning Hermes process is still alive. On Windows, if any handle to an exited owner process lingers, os.kill returns success — the reaper concludes the owner is alive and never cleans up the orphaned session directory. Stale session dirs accumulate forever.

The fix

One new public helper — gateway.status.pid_is_alive(pid, *, treat_permission_denied_as_alive=False) — with an internal _probe_pid_alive dispatcher.

• POSIX: delegate to os.kill(pid, 0). Behavior is byte-identical to the existing idiom.
• Windows: use the Win32 API directly via ctypes — OpenProcess(SYNCHRONIZE, ...) + WaitForSingleObject(handle, 0). os.kill is never consulted on Windows. A regression test (test_windows_bypasses_os_kill_entirely) codifies this invariant.

Design notes

• WaitForSingleObject over GetExitCodeProcess. MSDN explicitly warns that GetExitCodeProcess is unreliable for liveness because a process that legitimately exits with code STILL_ACTIVE == 259 reads as still running. WaitForSingleObject(handle, 0) has no such quirk.
• WinDLL("kernel32", use_last_error=True) + ctypes.get_last_error(). Without use_last_error=True, the default ctypes.windll.kernel32 does NOT reliably preserve Win32 LastError across the ctypes trampoline, and a stale LastError could be misread as ERROR_ACCESS_DENIED — exactly the bug this PR is fixing.
• treat_permission_denied_as_alive flag. Different call sites historically had different PermissionError semantics (some treated it as "stale/gone", others as "alive/don't touch"). The flag lets each site preserve its original behavior; see the mapping table below.
• _probe_pid_alive as the test mock point. Tests that previously monkey-patched status.os.kill to simulate process states now monkey-patch _probe_pid_alive, so the simulated state drives behavior on every platform regardless of which syscall the helper happens to use.

Call sites replaced

(*) Intentional behavior change. _stop_gateway_process originally caught only ProcessLookupError and let PermissionError propagate, crashing the wait loop. It now treats PermissionError as alive (the loop keeps waiting, then force-kills). This aligns with test_owner_pid_permission_error_treated_as_alive and the "don't reap what you can't see" principle used elsewhere in the codebase.

Test plan

• 15 new unit tests for pid_is_alive / _probe_pid_alive covering input validation, POSIX path, Windows path, permission-flag forwarding, and the Windows-doesn't-call-os.kill invariant.
• Existing tests/gateway/test_status.py tests updated to monkey-patch _probe_pid_alive (cross-platform) instead of status.os.kill (POSIX-only after this change).
• tests/tools/test_browser_orphan_reaper.py gained a _patch_liveness(mock_kill) helper that patches both os.kill and _probe_pid_alive from a single mock function; all 7 existing test sites migrated.
• All 86 tests in the directly-affected suites pass on Windows 11 (tests/gateway/test_status.py, test_runner_startup_failures.py, test_restart_drain.py, test_gateway_shutdown.py, and tests/tools/test_browser_orphan_reaper.py).

To verify locally

PYTHONPATH=. pytest tests/gateway/test_status.py tests/tools/test_browser_orphan_reaper.py tests/gateway/test_runner_startup_failures.py tests/gateway/test_restart_drain.py tests/gateway/test_gateway_shutdown.py

Manual reproduction on Windows

Before this PR:

$ hermes gateway run
OSError: [WinError 11] An attempt was made to load a program with an incorrect format

After this PR, the gateway starts cleanly; hermes gateway status from a second terminal no longer tears down the running gateway's PID file.

Related

This is the first of two Windows-targeted PRs. A follow-up will address the web-dashboard build path (web/package.json's sync-assets script uses Unix rm -rf / cp -r, and hermes_cli/main.py::_build_web_ui swallows stderr) — that's kept separate because it's a build-time shell-script portability issue rather than a runtime liveness-check issue.