Skip to content

fix: hermes update auto-restart kills in-process cron worker with no opt-out #6702

@r266-tech

Description

@r266-tech

Problem

hermes update unconditionally restarts all gateway processes after pulling new code (hermes_cli/main.py:3762-3850). There is no env var, CLI flag, or TTY check to opt out.

When hermes update is invoked from a cron job scheduled by hermes's own in-process cron scheduler (cron/scheduler.py tick loop + ThreadPoolExecutor worker thread runs inside the gateway Python process), the auto-restart SIGTERMs the gateway → the worker thread gets killed mid-flight → jobs.json post-run write never happens, state.db session row is missing, cron output file is incomplete.

The user sees: last_run_at stuck at yesterday even though git reflog shows the pull succeeded.

Repro

  1. Configure a cron job that runs hermes update
  2. Observe that after successful git pull, the gateway is killed
  3. jobs.json shows stale last_run_at — cron worker was killed before it could write completion state
  4. state.db sessions table has no row for the cron run — session was killed before persist

Evidence from git reflog: 2026-04-09 04:00:49 reset + 04:00:50 pull --ff-only (pull succeeded), then 04:01:09 Stopping gateway... (19s later, worker still running).

Proposed fix

Add an env var check before the auto-restart block:

# Skip gateway restart if opted out (e.g. hermes update from within
# a cron job whose worker thread runs inside the gateway process).
if os.environ.get("HERMES_SKIP_GATEWAY_RESTART"):
    return

# Auto-restart ALL gateways after update.
try:
    ...

This lets cron jobs set HERMES_SKIP_GATEWAY_RESTART=1 to safely run hermes update without killing their own host process.

~4 lines of code + 1 test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/cliCLI entry point, hermes_cli/, setup wizardcomp/cronCron scheduler and job managementtype/bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions