Likely duplicate of #13559 — both add pre-startup stale PID file cleanup in gateway/run.py after hard crashes. Also overlaps with #9703.

Likely duplicate of #13559

Comparison with Related PRs (#9703 and #13559)

Hi all — I wanted to cross-reference this PR with two other open efforts that address the same stale gateway.pid issue, in the interest of converging on the best combined approach.

cc @batumilove @john-livingston — would love your input!

PR #9703 (fix(gateway): clear stale pid file before service start) by @john-livingston takes a clean, minimal approach by adding ExecStartPre=-/bin/rm -f to the systemd service templates. This is elegant in its simplicity — it sidesteps the race condition entirely by removing the file before Python even starts. The limitation is that it only covers systemd-managed deployments, leaving CLI and container starts without the same protection.

PR #13559 (fix(gateway): resolve stale PID file blocking startup after forced kill) by @batumilove does excellent work identifying a logic bug in get_running_pid() where _looks_like_gateway_process() and _record_looks_like_gateway() can return contradictory results, causing the stale-file case to go unhandled. Their fix — reading /proc/<pid>/cmdline to verify the live process — is thorough and works across all deployment types. Really nice root-cause analysis.

This PR (#13658) adds a pre-startup os.kill(pid, 0) cleanup block as a lightweight fallback before the existing guard runs. It's designed to be purely additive and deployment-agnostic.

How they relate:

These approaches are complementary rather than competing:

• fix(gateway): clear stale pid file before service start #9703 covers the systemd case at the OS level (belt)
• fix(gateway): resolve stale PID file blocking startup after forced kill #13559 fixes the underlying logic bug in the duplicate-instance guard (the real root cause)
• fix: force-clean stale PID file on gateway startup #13658 provides a pre-startup cleanup fallback for cases where/proc may be unreadable or where the gateway isn't managed by systemd (suspenders)

I think #13559 addresses the most important issue — the logic contradiction in get_running_pid() — and would be great to get merged. If there's interest, I'd be happy to incorporate that fix here, or rebase this PR on top of #13559 to provide the defense-in-depth layer.

One small note: #13559 also includes a GLM/Zhipu error 1213 fix, which might be cleaner as a separate PR to keep review scope focused.

Would love to hear thoughts from both of you on whether a combined approach makes sense. Great work from everyone on tackling this issue from different angles.

Thanks for the thorough cross-reference @natehale — really helpful to have all three approaches laid out together.

Having now read the actual diffs for #13559 and #13709, I think the picture is clearer than it first appeared. They address genuinely different failure modes:

• fix(gateway): clear stale PID file from crashed gateway on startup (closes #13655) #13709 fixes _cleanup_invalid_pid_path() — the ownership guard in remove_pid_file() (file_pid != os.getpid()) correctly protects the --replace handoff race, but wrongly fires when the process is dead: a crashed gateway can never satisfy that condition, so the stale file is silently preserved. Direct unlink() is correct here since dead-PID has already been confirmed. The regression tests make this airtight.
• fix(gateway): resolve stale PID file blocking startup after forced kill #13559 fixes the PID-reuse case in get_running_pid() — when a live unrelated process happens to have the same PID, _looks_like_gateway_process() returns False but _record_looks_like_gateway() returns True, causing a false "still running" result. The cmdline cross-check breaks that tie. Worth noting: fix(gateway): resolve stale PID file blocking startup after forced kill #13559 also bundles a GLM/Zhipu error 1213 fix in gateway/run.py — that part is unrelated and would be cleaner as a separate PR.

Both should merge; neither makes the other redundant.

As for #9703: it operates at a completely different layer (hermes_cli/gateway.py, not gateway/status.py) and covers a scenario none of the Python-level fixes address — when Python can't start cleanly at all. The ExecStartPre=-/bin/rm -f line runs before Python is invoked, so even if get_running_pid() logic is perfect, a stale file left after a hard kill between systemd restarts is cleared unconditionally. It's the OS-level belt to the Python-level suspenders. No conflict with any of the above.

Happy to rebase #9703 on top of whatever merges first.