Skip to content

Telegram isolated ingress copies large process.env in queue hot path #94571

Description

@kaka-srp

Summary

In OpenClaw 2026.6.6, Telegram isolated polling ingress can saturate the main gateway event loop when the process environment is large. The hot path is the durable channel ingress queue state DB opener:

// src/channels/message/ingress-queue.ts
openOpenClawStateDatabase({
  env: stateDir ? { ...process.env, OPENCLAW_STATE_DIR: stateDir } : process.env,
});

When stateDir is set, this copies every key in process.env. In Kubernetes pods with enableServiceLinks left at the default, the pod can inherit thousands of service environment variables. Telegram isolated polling drains its local spool every ~500ms, so this copy happens repeatedly even when the actual SQLite queries are cheap.

Impact

The symptom is high CPU / event-loop delay in the OpenClaw gateway, with delayed or missing replies on other channels as collateral damage. The issue is not Telegram API latency or SQLite itself; it is repeatedly enumerating a very large process.env before opening the state DB.

Production evidence

Observed on a Kubernetes deployment running OpenClaw 2026.6.6 with Telegram configured and isolated polling ingress enabled.

Before mitigation:

  • Object.keys(process.env).length: 9274
  • { ...process.env } once: 917ms to 1192ms
  • createChannelIngressQueue(...).listPending() once: ~982ms
  • listClaims() once: ~816ms
  • recoverStaleClaims() once: ~869ms

Control checks:

  • Direct SQLite was fast: 10000 SELECT count(*) FROM channel_ingress_events took ~118ms.
  • Direct OpenClaw state DB + Kysely queries took ~0-2ms.
  • Disabling Kubernetes service links reduced env keys to ~60 and made spread effectively 0ms.

Local code patch experiment:

  • Replacing the spread with a prototype overlay reduced queue wrapper calls dramatically.
  • First listPending() call was ~47ms, then 0-1ms warm.
  • listPending() x1000 was ~270ms, listClaims() x1000 ~140ms, recoverStaleClaims() x1000 ~147ms.

Reproduction

One way to reproduce without a full Kubernetes cluster is to run the gateway or the queue tests with a large environment and a queue state dir override:

  1. Create many environment variables, similar to Kubernetes service env injection:
for (let i = 0; i < 10000; i++) {
  process.env[`KUBERNETES_SERVICE_${i}`] = "tcp://10.0.0.1:443";
}
  1. Create a channel ingress queue with stateDir set and call drain-like operations repeatedly:
const queue = createChannelIngressQueue({
  channelId: "telegram",
  accountId: "default",
  stateDir,
});

await queue.recoverStaleClaims({ staleMs: 0 });
await queue.listClaims();
await queue.listPending({ limit: 1000, orderBy: "id" });
  1. Compare the runtime with the current spread implementation versus a non-copying overlay:
const env = Object.assign(Object.create(process.env), {
  OPENCLAW_STATE_DIR: stateDir,
});

Expected behavior

Opening a channel ingress queue for a custom stateDir should not enumerate and copy the full host process environment on every queue operation. It only needs OPENCLAW_STATE_DIR to override state resolution while preserving inherited lookups such as HOME / OPENCLAW_HOME.

Suggested fix

Avoid { ...process.env } in the ingress queue hot path. Use a lightweight overlay object or pass a direct DB path/state dir through the state DB API.

A minimal compatible fix is:

function createStateDirEnv(stateDir: string, baseEnv = process.env) {
  return Object.assign(Object.create(baseEnv), {
    OPENCLAW_STATE_DIR: stateDir,
  });
}

This keeps inherited env lookups working but avoids enumerating thousands of Kubernetes service env vars.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions