-
-
Notifications
You must be signed in to change notification settings - Fork 80.3k
Telegram isolated ingress copies large process.env in queue hot path #94571
Copy link
Copy link
Closed
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
No fields configured for issues without a type.
Summary
In OpenClaw 2026.6.6, Telegram isolated polling ingress can saturate the main gateway event loop when the process environment is large. The hot path is the durable channel ingress queue state DB opener:
When
stateDiris set, this copies every key inprocess.env. In Kubernetes pods withenableServiceLinksleft at the default, the pod can inherit thousands of service environment variables. Telegram isolated polling drains its local spool every ~500ms, so this copy happens repeatedly even when the actual SQLite queries are cheap.Impact
The symptom is high CPU / event-loop delay in the OpenClaw gateway, with delayed or missing replies on other channels as collateral damage. The issue is not Telegram API latency or SQLite itself; it is repeatedly enumerating a very large
process.envbefore opening the state DB.Production evidence
Observed on a Kubernetes deployment running OpenClaw 2026.6.6 with Telegram configured and isolated polling ingress enabled.
Before mitigation:
Object.keys(process.env).length: 9274{ ...process.env }once: 917ms to 1192mscreateChannelIngressQueue(...).listPending()once: ~982mslistClaims()once: ~816msrecoverStaleClaims()once: ~869msControl checks:
10000 SELECT count(*) FROM channel_ingress_eventstook ~118ms.Local code patch experiment:
listPending()call was ~47ms, then 0-1ms warm.listPending()x1000 was ~270ms,listClaims()x1000 ~140ms,recoverStaleClaims()x1000 ~147ms.Reproduction
One way to reproduce without a full Kubernetes cluster is to run the gateway or the queue tests with a large environment and a queue state dir override:
stateDirset and call drain-like operations repeatedly:Expected behavior
Opening a channel ingress queue for a custom
stateDirshould not enumerate and copy the full host process environment on every queue operation. It only needsOPENCLAW_STATE_DIRto override state resolution while preserving inherited lookups such asHOME/OPENCLAW_HOME.Suggested fix
Avoid
{ ...process.env }in the ingress queue hot path. Use a lightweight overlay object or pass a direct DB path/state dir through the state DB API.A minimal compatible fix is:
This keeps inherited env lookups working but avoids enumerating thousands of Kubernetes service env vars.