fix(gateway): clean stale PID file before O_EXCL write#25569
Closed
zccyman wants to merge 1 commit into
Closed
Conversation
4d88964 to
c1a3596
Compare
Contributor
|
This looks implemented on current main now. Automated hermes-sweeper review found the stale gateway PID restart case already covered by the gateway startup/status path. Evidence:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #13511
After an abnormal gateway exit (SIGKILL, crash, OOM, terminal disconnect), the
gateway.pidfile remains on disk. On next gateway start,write_pid_file()usesO_CREAT | O_EXCLwhich raisesFileExistsError— causing the gateway to exit with "PID file race lost to another gateway instance" even though the old process is dead.Root Cause
write_pid_file()ingateway/status.pydoes not check whether the process recorded in a pre-existing PID file is still alive before attempting the atomicO_EXCLcreate. TheFileExistsErroris unconditionally re-raised, blocking restart.Fix
Before the
os.open(path, O_CREAT | O_EXCL | ...)call, add a stale PID cleanup step:os.kill(pid, 0)ProcessLookupError(process dead), remove the stale fileO_EXCLcreationThis preserves the atomic race behavior for truly concurrent starts while fixing the stale-file blocking case.
Testing
Added 2 new tests in
tests/gateway/test_status.py:test_write_pid_file_cleans_stale_pid_from_dead_process— stale PID file is cleaned, write succeedstest_write_pid_file_preserves_pid_file_for_live_process— live process PID file preserved,FileExistsErrorraisedAll 13
TestGatewayPidStatetests pass.