Skip to content

Webhook plugin re-registers repeatedly whenever event loop saturates (dreaming, cron catchup, Control UI polling) #75298

@balaji1968-kingler

Description

@balaji1968-kingler

Version: 2026.4.27 (also reproduced on 2026.4.26)
Setup: Docker container, Windows 11 host, Compose stack, single webhook route (ai-news)

Summary

The webhooks plugin re-registers its route repeatedly (5–10x per gateway session) whenever the Node.js event loop saturates. This is not limited to cold start — it reproduces via at least three distinct triggers observed in the same gateway session.

Reproduction scenarios

All three occurred in a single overnight gateway session (same uptime, no restart):

Scenario 1 — Startup with missed cron jobs:
```
20:45:05 — [webhooks] registered route ai-news (normal startup)
20:46:45 — [webhooks] registered route ai-news ← re-registration #2
20:47:20 — [webhooks] registered route ai-news ← re-registration #3
20:47:32 — [webhooks] registered route ai-news ← re-registration #4
20:47:44 — [webhooks] registered route ai-news ← re-registration #5
20:47:57 — [webhooks] registered route ai-news ← re-registration #6
20:47:58 — [diagnostic] eventLoopDelayMaxMs=51506ms eventLoopUtilization=1.0 ← CPU fully saturated
20:47:58 — [whatsapp] Web connection closed (status 408)
```

Scenario 2 — Control UI polling storm:
```
21:32:52–21:33:11 — rapid sessions.list + node.list calls from Control UI (10 calls in 19s)
21:33:11 — [webhooks] registered route ai-news ← re-registration
21:33:12 — [diagnostic] eventLoopDelayMaxMs=12884ms eventLoopUtilization=0.875
21:33:12 — [ws] node.list resolved in 14131ms
```

Scenario 3 — memory-core dreaming run:
```
02:00:16 — [diagnostic] liveness warning (event loop delay)
02:00:19 — [webhooks] registered route ai-news ← re-registration
02:00:38 — [webhooks] registered route ai-news ← re-registration
02:00:39 — [whatsapp] Web connection closed (status 408)
02:00:39 — [ws] node.list resolved in 18810ms
02:00:45 — [plugins] memory-core: normalized recall artifacts before dreaming
02:01:02 — [webhooks] registered route ai-news ← re-registration
```

Idle baseline

Even during quiet periods between these events, the event loop shows persistent background delay:
```
eventLoopDelayMaxMs consistently 5000–6000ms across multiple 30s diagnostic intervals
eventLoopUtilization ~0.3–0.5 at idle
```

Pattern

Re-registrations always coincide with or immediately follow a [diagnostic] liveness warning with elevated eventLoopDelayMaxMs or eventLoopUtilization. No config reload, channel restart, plugin bootstrap, or session reset lines are visible between consecutive re-registrations.

Hypothesis

The webhook plugin's register() is being called each time the plugin runtime reloads or re-initialises a session surface under event loop pressure. The root trigger is event loop saturation, not any specific feature — cron catchup, dreaming, and Control UI polling all reproduce it independently.

Impact

  • WhatsApp 408 drops correlating with saturation events
  • Channel node.list and sessions.list WS calls taking 14–18s during cascade
  • Unclear whether duplicate route registrations cause handler duplication or are safely idempotent

Question

Is webhooks plugin register() expected to be called more than once per gateway lifecycle? If not, what session/plugin lifecycle event is triggering re-registration under event loop load? Is there a guard that should be preventing this?

Version: 2026.4.27 | Docker | Windows 11 host | Compose stack

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions