Fixes #28181 — kanban workers stuck in running state forever after gateway SIGTERM.

Summary

The single-query signal handler in cli.py (_signal_handler_q) raises KeyboardInterrupt on SIGTERM/SIGHUP. For interactive hermes chat -q that unwinds the main thread cleanly. For kanban workers spawned by the dispatcher, the worker process is likely to have a non-daemon thread alive (terminal _wait_for_process drains, custom plugin threads, etc.). With KeyboardInterrupt only the main thread unwinds; the non-daemon thread keeps the process alive after the gateway restarts, and _pid_alive returns True forever → task stays running → claim extended forever → manual hermes kanban block <id> required.

The fix: when HERMES_KANBAN_TASK is set (dispatcher-spawned worker), flush logging + stdout/stderr and call os._exit(0) instead of raising KeyboardInterrupt. The kernel reclaims the PID immediately, _pid_alive's existing zombie-state check flips the task to crashed, and detect_crashed_workers re-spawns it on the next tick. No manual recovery needed.

Live reproduction + verification

Verified end-to-end with a synthetic worker that holds a non-daemon thread blocked on threading.Event().wait() (matches the shape of _wait_for_process blocking on a child subprocess):

The repro and the fix-path test both live in tests/hermes_cli/test_signal_handler_kanban_worker.py as 3 regression cases.

Design notes

• Env-gated: the os._exit path is gated specifically on os.environ.get("HERMES_KANBAN_TASK"). Interactive hermes chat -q (no env var) keeps the original KeyboardInterrupt behavior — unchanged. The env var is the canonical "this process is a dispatcher-spawned worker" signal (set in hermes_cli/kanban_db.py:5921).
• SIGALRM deadman: before flushing, signal.alarm(2) is armed with a handler that calls os._exit(0). Reporter measured flush in <1ms under normal conditions; the alarm exists as a failsafe in case logging's stream handlers have a pathological blocking close. Never fires in practice; pure belt-and-suspenders.
• Flush order: logging.shutdown() → sys.stdout.flush() → sys.stderr.flush() → os._exit(0). Final telemetry trace preserved before the kernel reaps.
• Non-daemon-thread invariant preserved: we don't change anything about how kanban workers spawn threads. Workers can continue to use non-daemon threads freely; the signal handler's job is to ensure SIGTERM still kills the process even when those threads exist.

Why not raise KeyboardInterrupt better

Reviewer feedback on the original PR draft (@hanselhansel) flagged operational concerns about os._exit(0) bypassing Python cleanup. The reporter (@andrewhosf) verified flush + 24h stability before re-proposing. The version landing here matches the validated design: explicit flush + SIGALRM deadman + env-gated to dispatcher-spawned workers only.

A "cleaner unwind" alternative (e.g., signal the non-daemon threads to exit, join them, then sys.exit) is genuinely harder because we don't own all the non-daemon thread surface (custom plugins can spawn anything they want). The signal-handler contract in production code has to terminate the process from outside the thread surface, which is what os._exit does.

Validation

• tests/hermes_cli/test_signal_handler_kanban_worker.py: 3/3 pass
  • End-to-end SIGTERM with HERMES_KANBAN_TASK set → process dies in <2s
  • End-to-end SIGTERM without HERMES_KANBAN_TASK → original KeyboardInterrupt path, unchanged
  • Source-level invariant test catching future refactors that drop the env-gated exit
• tests/tools/test_kanban_tools.py + tests/hermes_cli/test_kanban_db.py + tests/hermes_cli/test_kanban_core_functionality.py: 449/449 pass

Credit

• @andrewhosf — identified the bug, diagnosed the KeyboardInterrupt-vs-non-daemon-thread root cause, designed the os._exit fix concept with flush + SIGALRM deadman, ran 24h stability validation. Co-authored trailer on the commit.
• @hanselhansel — operational-risk review of the original draft (state corruption, telemetry loss, stability metrics) that drove the flush-before-exit + deadman design that's actually shipping here.

Infographic