fix(telegram): webhook hang - tests and fix by huntharo · Pull Request #26933 · openclaw/openclaw

huntharo · 2026-02-25T22:58:20Z

Summary

Describe the problem and fix in 2–5 bullets:

Problem: Telegram webhook request bodies could be consumed before Grammy processed them, causing first-request failures, delayed-read failures, and truncation/malformed payload behavior (issue [Bug]: Telegram Webhook - 1-2 minute delay of message receipt on startup - Due to recent change #26156). Additionally, the top two commits de-register the webhook on gateway restart/config reload so that Telegram will not get an error calling the hook during restart, which then causes a delay in the ability to send messages when the gateway comes back up (~1 minute delay).
Why it matters: incoming Telegram updates can fail on startup or under delayed/large-body conditions, leading to dropped messages and webhook instability; additionally it appears that the delays would happen at random while using the webhook but polling was not impacted.
What changed: switched Telegram webhook handling to webhookCallback(..., "callback"), pre-initialized bot startup, read request JSON exactly once via readJsonBodyWithLimit, mapped body-read failures to explicit HTTP statuses, and added regression tests for delayed reads, multi-request behavior, near-limit payloads, and >1MB rejection. The webhook is deregistered during restarts, then re-registered when the gateway comes back up.
What did NOT change (scope boundary): no webhook path/secret configuration model changes, no increase to max body limit (still 1MB), no protocol changes outside Telegram webhook handling. The webhook should also be deregistered on SIGTERM graceful shutdown, but that does not appear to be reaching this channel and is out of scope.

AI Assisted: Used Codex. Took many, many, many iterations to zero in on the minimal change. Lots of manual testing to confirm that the real issue was fixed instead of just causing tests to pass. This took many hours of work even utilizing Codex.

Note - Avoiding Bug in `grammy`

grammy has a bug in the http handler that results in an uncatchable exception and unresolved promise when the webhook contents are discarded by the guard calling end before the grammy reader has read any of the data. It's not possible to continue to use the http handler until that bug is fixed, but it really shouldn't be used even then as consolidating the length check and having only one stream reader is much simpler. The bug fix PR is here:

grammyjs/grammY#872

Change Type (select all)

Scope (select all touched areas)

Linked Issue/PR

User-visible / Behavior Changes

Telegram webhook body parsing is now deterministic and single-pass before callback dispatch, eliminating first-request/body-read race conditions.
Invalid/oversized/timeout webhook bodies return explicit status codes (400, 408, 413) instead of surfacing as generic handler failures.
Startup now fails fast if bot initialization (getMe) fails, instead of deferring initialization side effects to first incoming webhook request.

Security Impact (required)

New permissions/capabilities? (No)
Secrets/tokens handling changed? (No)
New/changed network calls? (Yes)
Command/tool execution surface changed? (No)
Data access scope changed? (No)
If any Yes, explain risk + mitigation:
- bot.init()/getMe is now executed explicitly during webhook startup. Risk: startup can fail earlier when Telegram API/token is invalid or unreachable. Mitigation: explicit error logging, startup rejection is immediate and observable, and behavior is covered by tests.

Repro + Verification

Environment

OS: macOS (local dev machine)
Runtime/container: Node.js v24.13.0, pnpm, Vitest v4.0.18
Model/provider: N/A
Integration/channel (if any): Telegram webhook
Relevant config (redacted): webhook secret, webhook path, default body limits/timeouts

Steps

Run pnpm test src/telegram/webhook.test.ts on a baseline with the body-read race behavior (revert the commits up to the last test before the fix
Observe failures in delayed-read/body-read scenarios.
Apply this branch and rerun pnpm test src/telegram/webhook.test.ts.
See [Bug]: Telegram Webhook - 1-2 minute delay of message receipt on startup - Due to recent change #26156 for VIDEO of how to reproduce this in a live OpenClaw Gateway with Telegram Webhook

Expected

All Telegram webhook regression tests pass, including delayed reads, sequential requests, near-limit payloads, and >1MB rejection.

Actual

With this branch, the webhook test suite passes and validates stable body handling across those scenarios.
With the fix reverted several of the tests fail or emit uncatchable errors

Evidence

Attach at least one:

Failing test/log before + passing after
Trace/log snippets
Screenshot/recording
Perf numbers (if relevant)

Video

See #26156 for video of the issue before the fix. Will attach a video of the fix in action shortly.

Failing Tests - Gateway Restart WebHook Deregistration

After Fix

Fix Reverted

Failing Tests - Stream Reading

After Fix

Fix Reverted

Failing Logs with Fix Reverted

pnpm test src/telegram/webhook.test.ts


> openclaw@2026.2.25 test /Users/huntharo/.codex/worktrees/44b6/openclaw
> node scripts/test-parallel.mjs src/telegram/webhook.test.ts


 RUN  v4.0.18 /Users/huntharo/.codex/worktrees/44b6/openclaw

 ✓ src/telegram/webhook.test.ts (9 tests) 256ms
   ✓ startTelegramWebhook (9)
     ✓ starts server, registers webhook, and serves health 13ms
     ✓ invokes webhook handler on matching path 6ms
     ✓ rejects startup when webhook secret is missing 1ms
     ✓ keeps webhook payload readable when callback delays body read 54ms
     ✓ keeps webhook payload readable across multiple delayed reads 105ms
     ✓ processes a second request after first-request delayed-init data loss 3ms
     ✓ handles near-limit payload with random chunk writes and event-loop yields 66ms
     ✓ handles near-limit payload written in a single request write 7ms
     ✓ rejects payloads larger than 1MB before invoking webhook handler 1ms

 Test Files  1 passed (1)
      Tests  9 passed (9)
   Start at  18:02:10
   Duration  483ms (transform 130ms, setup 106ms, import 64ms, tests 256ms, environment 0ms)


openclaw on  telegram-webhook-hang-tests-and-fix [$⇕] is 📦 v2026.2.25 via  v24.13.0 via 🐍 v3.14.3 
❯ pnpm test src/telegram/webhook.test.ts


> openclaw@2026.2.25 test /Users/huntharo/.codex/worktrees/44b6/openclaw
> node scripts/test-parallel.mjs src/telegram/webhook.test.ts


 RUN  v4.0.18 /Users/huntharo/.codex/worktrees/44b6/openclaw

 ❯ src/telegram/webhook.test.ts (9 tests | 5 failed) 16511ms
   ❯ startTelegramWebhook (9)
     ✓ starts server, registers webhook, and serves health 13ms
     ✓ invokes webhook handler on matching path 4ms
     ✓ rejects startup when webhook secret is missing 1ms
     × keeps webhook payload readable when callback delays body read 134ms
     × keeps webhook payload readable across multiple delayed reads 129ms
     × processes a second request after first-request delayed-init data loss 8133ms
     × handles near-limit payload with random chunk writes and event-loop yields 77ms
     × handles near-limit payload written in a single request write 8007ms
     ✓ rejects payloads larger than 1MB before invoking webhook handler 12ms

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Failed Tests 5 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 FAIL  src/telegram/webhook.test.ts > startTelegramWebhook > keeps webhook payload readable when callback delays body read
AssertionError: expected 500 to be 200 // Object.is equality

- Expected
+ Received

- 200
+ 500

 ❯ src/telegram/webhook.test.ts:428:26
    426|       );
    427| 
    428|       expect(res.status).toBe(200);
       |                          ^
    429|       await expect(res.text()).resolves.toBe(payload);
    430|     } finally {

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/5]⎯

 FAIL  src/telegram/webhook.test.ts > startTelegramWebhook > keeps webhook payload readable across multiple delayed reads
AssertionError: expected 500 to be 200 // Object.is equality

- Expected
+ Received

- 200
+ 500

 ❯ src/telegram/webhook.test.ts:488:28
    486|           5_000,
    487|         );
    488|         expect(res.status).toBe(200);
       |                            ^
    489|       }
    490| 

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/5]⎯

 FAIL  src/telegram/webhook.test.ts > startTelegramWebhook > processes a second request after first-request delayed-init data loss
 FAIL  src/telegram/webhook.test.ts > startTelegramWebhook > handles near-limit payload written in a single request write
Error: webhook post timed out after 8000ms
 ❯ Timeout.<anonymous> src/telegram/webhook.test.ts:237:20
    235| 
    236|     const timeout = setTimeout(() => {
    237|       finishReject(new Error(`webhook post timed out after ${params.timeoutMs ?? 15_000}ms`));
       |                    ^
    238|       req.destroy();
    239|     }, params.timeoutMs ?? 15_000);

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[3/5]⎯

 FAIL  src/telegram/webhook.test.ts > startTelegramWebhook > handles near-limit payload with random chunk writes and event-loop yields
AssertionError: expected 182665 to be 1044480 // Object.is equality

- Expected
+ Received

- 1044480
+ 182665

 ❯ src/telegram/webhook.test.ts:589:41
    587|       expect(response.statusCode).toBe(200);
    588|       expect(capturedBodies).toHaveLength(1);
    589|       expect(capturedBodies[0]?.length).toBe(payload.length);
       |                                         ^
    590|       expect(sha256(capturedBodies[0] ?? "")).toBe(sha256(payload));
    591|     } finally {

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[4/5]⎯

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

Vitest caught 1 unhandled error during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Uncaught Exception ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
SyntaxError: Unexpected end of JSON input
 ❯ IncomingMessage.<anonymous> node_modules/.pnpm/grammy@1.40.0/node_modules/grammy/out/convenience/frameworks.js:207:34
 ❯ Object.onceWrapper node:events:622:28
 ❯ IncomingMessage.emit node:events:520:35
 ❯ endReadableNT node:internal/streams/readable:1701:12
 ❯ processTicksAndRejections node:internal/process/task_queues:89:21

This error originated in "src/telegram/webhook.test.ts" test file. It doesn't mean the error was thrown inside the file itself, but while it was running.
The latest test that might've caused the error is "processes a second request after first-request delayed-init data loss". It might mean one of the following:
- The error was thrown, while Vitest was running this test.
- If the error occurred after the test had been completed, this was the last documented test before it was thrown.
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯


 Test Files  1 failed (1)
      Tests  5 failed | 4 passed (9)
     Errors  1 error
   Start at  18:03:05
   Duration  16.75s (transform 132ms, setup 113ms, import 64ms, tests 16.51s, environment 0ms)

 ELIFECYCLE  Test failed. See above for more details.

Human Verification (required)

What you personally verified (not just CI), and how:

Verified scenarios:
- pnpm test src/telegram/webhook.test.ts
- delayed callback read behavior
- multi-request sequencing after first-request delay
- near-limit payload handling (chunked + single-write)
- 1MB payload rejection before handler invocation
- Manual
  - Restart gateway / immediately able to message on Telegram with no delay using webhook
  - Tested dozens of times
  - See video on linked issue for steps to reproduce before and after this fix
Edge cases checked:
- random chunk boundaries + event-loop yields
- explicit oversize rejection path
What you did not verify:
- long-running production Telegram traffic under real network jitter/load (in progress, using this for my main gateway)

Compatibility / Migration

Backward compatible? (Yes)
Config/env changes? (No)
Migration needed? (No)
If yes, exact upgrade steps:

Failure Recovery (if this breaks)

How to disable/revert this change quickly:
- Revert this PR
Files/config to restore:
- src/telegram/webhook.ts
- src/telegram/webhook.test.ts (if test expectations need to be reverted together)
Known bad symptoms reviewers should watch for:
- webhook 500 responses on valid payloads
- first Telegram updates failing after startup
- delayed-read tests regressing

Risks and Mitigations

List only real risks for this PR. Add/remove entries as needed. If none, write None.

Risk: callback-mode integration could drift from Grammy expectations in future updates.
- Mitigation: regression tests cover delayed reads, sequential requests, near-limit bodies, and oversize rejection.
Risk: startup now depends on immediate Telegram API reachability.
- Mitigation: fail-fast startup behavior is explicit, logged, and easier to diagnose than latent first-request failures.

Update mocks

Takhoffman · 2026-02-26T02:57:08Z

PR #26933 - fix(telegram): webhook hang - tests and fix (#26933)

Merged via squash.

Merge commit: ee594e2
Verified: pnpm build, pnpm check, pnpm test:macmini
Changes made:
M\tCHANGELOG.md
Why these changes were made:
Added the required changelog entry for the Telegram webhook hang fix before merge, per autoland policy.
Changelog: CHANGELOG.md updated=true required=true opt_out=false

Thanks @huntharo!

KnorpelSenf · 2026-02-26T08:04:58Z

grammy has a bug in the http handler

We have released the patch for grammY. If you use v1.40.1, this no longer happens.

…huntharo