Skip to content

Background process completion and watch notifications can duplicate #15276

@realraelrr

Description

@realraelrr

Bug Description

Hermes can deliver duplicate agent-facing background-process notifications for the same process when notify_on_complete and watch_patterns are both involved.

The duplicate path is not specific to one adapter or one command. It can affect any background process where completion delivery and watch-pattern delivery both produce synthetic [SYSTEM: ...] events for the same process session.

Steps to Reproduce

  1. Start a Hermes background process with notify_on_complete enabled.
  2. Configure one or more watch_patterns that can match the process output.
  3. Let the process produce matching output and then exit.
  4. Observe the gateway delivering an agent-facing completion notification.
  5. A later watch notification for the same process can still be injected, interrupting the agent even though the completion notification was already delivered.

Expected Behavior

  • Once the gateway successfully injects a completion notification into the agent thread, that process completion should be treated as consumed.
  • Later watch-pattern notifications for the same completed process should not interrupt the agent.
  • Existing wait, poll, and read_log completion-consumption behavior should remain unchanged.
  • Watch patterns should still fire for active processes before the completion has been consumed.

Actual Behavior

  • wait, poll, and read_log already suppress redundant completion delivery by tracking _completion_consumed.
  • Gateway-injected notify_on_complete messages were not marking the completion as consumed.
  • A watcher path could therefore still inject a related watch notification for the same process after the agent had already received the completion notification.
  • In practice this can show up as repeated synthetic system notifications or follow-up messages indicating the notification was already handled.

Affected Component

  • Gateway notification delivery
  • Background process registry / tools
  • Gateway adapters that receive synthetic process events

Messaging Platform

Gateway-related; not specific to Telegram, Discord, Slack, or WhatsApp.

Debug Report

Not available. This was identified from the background-process notification flow and covered with targeted regression tests in #15277.

Operating System

N/A; runtime behavior is platform-independent.

Python Version

N/A.

Hermes Version

Current main before #15277.

Root Cause Analysis

ProcessRegistry tracks _completion_consumed so wait, poll, and read_log can avoid delivering the same completion more than once. However, successful gateway injection of an agent-facing completion notification did not update that consumed state.

Because the process was still considered unconsumed by the registry, _inject_watch_notification could later deliver a watch-pattern event for the same completed process.

Proposed Fix

Implemented in #15277:

  • Add ProcessRegistry.mark_completion_consumed(session_id).
  • Call it after successful gateway injection of a notify_on_complete synthetic event.
  • Skip watch notification injection when the process completion is already consumed.
  • Add regression coverage for gateway completion injection, watch suppression, and direct registry marking.

Validation

The PR validates the fix with:

uv run --extra dev pytest tests/tools/test_notify_on_complete.py tests/gateway/test_internal_event_bypass_pairing.py tests/tools/test_watch_patterns.py
python3.11 -m py_compile gateway/run.py tools/process_registry.py

PR

Fix: #15277

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/gatewayGateway runner, session dispatch, deliverycomp/toolsTool registry, model_tools, toolsetstype/bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions