fix(stream-consumer): use UTF-16 length for Telegram message splitting (salvage of #11170) by teknium1 · Pull Request #23455 · NousResearch/hermes-agent

teknium1 · 2026-05-10T23:18:20Z

Summary

Telegram's 4096-character message limit is measured in UTF-16 code units, not Python codepoints. Emoji- or CJK-heavy responses with surrogate-pair characters could blow past the limit silently — len() says 3000, utf16_len() says 5500 — and Telegram would reject the send or render \x00 (black square) artifacts.

After this PR, the streaming consumer measures length using each adapter's native message_len_fn. Telegram declares utf16_len; everyone else inherits the default len. TelegramAdapter.send() already used utf16_len for the non-streaming path; this closes the asymmetry.

Changes

gateway/platforms/base.py: add BasePlatformAdapter.message_len_fn property defaulting to len.
gateway/platforms/telegram.py: override message_len_fn to return utf16_len.
gateway/stream_consumer.py: read adapter.message_len_fn (gated on isinstance(BasePlatformAdapter)) and thread it through _safe_limit math, overflow detection, truncate_message, _split_text_chunks, and _send_fallback_final chunking. Use _custom_unit_to_cp to convert the UTF-16 budget back to a codepoint index for rfind-based split-point selection.
tests/gateway/test_stream_consumer.py: new TestUtf16OverflowDetection class with two regression tests.
scripts/release.py: AUTHOR_MAP entry for Freeman-Consulting.

Improvements during salvage

Cleaned up the len_fn lookup: replaced the original PR's two duplicated try/except + delayed-import + isinstance(_BPA) blocks with a single module-level import from gateway.platforms.base import BasePlatformAdapter as _BasePlatformAdapter and a clean ternary at each callsite. The isinstance gate is necessary because test MagicMock adapters' auto-attribute message_len_fn is a mock object, not a callable; falling back to len for those preserves the existing test suite.
Dropped the if _len_fn is not len micro-optimization guards around _custom_unit_to_cp. The helper itself short-circuits via if len_fn(s) <= budget: return len(s), so calling it unconditionally is safe and reads cleaner.
Added an inline comment at the buffer_threshold trigger noting the intentional codepoint-based length (it's a debounce heuristic — "edit roughly every N visible characters" — not a platform-limit check, so leaving it on plain len is correct).
Added regression tests: the original PR shipped no tests for the UTF-16 path. New TestUtf16OverflowDetection covers the emoji-overflow case end-to-end, asserting truncate_message was called with len_fn=utf16_len.

Validation

tests/gateway/test_stream_consumer.py + tests/gateway/test_stream_consumer_thread_routing.py: 95/95 pass in 3.9s.

Interaction with PR #19537

Orthogonal and complementary. #19537 makes Telegram's edit_message return success=False with error="message_too_long" instead of silently truncating on overflow. This PR prevents the overflow in the first place. Together = belt-and-suspenders.

Closes the bug behind #11170.
Authored by @Freeman-Consulting; cleanup commits and tests follow.

The stream consumer measured message length using Python's len() (Unicode code points), but Telegram's actual limit is in UTF-16 code units. This caused messages with supplementary characters (emoji, CJK, etc.) to exceed Telegram's 4096-character limit, resulting in truncated messages with formatting artifacts. Changes: - Add message_len_fn property to BasePlatformAdapter (defaults to len) - Override in TelegramAdapter to return utf16_len - Stream consumer uses adapter.message_len_fn for: - safe_limit calculation - overflow detection - truncate_message calls - split point calculation (via _custom_unit_to_cp) - fallback final send chunking Fixes truncated messages with black square artifacts on Telegram when the model generates responses containing multi-byte Unicode characters.

New TestUtf16OverflowDetection class covers two scenarios: - test_emoji_text_exceeding_utf16_limit_triggers_overflow_split: feeds 2200 emoji codepoints (4400 UTF-16 units) — under Telegram's codepoint-equivalent limit but over its UTF-16 limit. Asserts truncate_message was called with len_fn=utf16_len, confirming the consumer detected the overflow. - test_codepoint_only_adapter_falls_back_to_len: documents that adapters which don't subclass BasePlatformAdapter (or test MagicMocks) fall back to plain len for backwards compat. The contributor's PR shipped no tests for the UTF-16 path.

github-actions · 2026-05-10T23:19:25Z

🔎 Lint report: `salvage/pr-11170` vs `origin/main`

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 8085 on HEAD, 8085 on base (➖ 0)

🆕 New issues (3):

Rule	Count
`invalid-argument-type`	3

First entries

run_agent.py:7160: [invalid-argument-type] invalid-argument-type: Argument to function `build_anthropic_client` is incorrect: Expected `str`, found `str | dict[Unknown, Unknown] | Any | ... omitted 3 union elements`
run_agent.py:13287: [invalid-argument-type] invalid-argument-type: Argument to function `len` is incorrect: Expected `Sized`, found `(str & ~AlwaysFalsy) | (dict[Unknown, Unknown] & ~AlwaysFalsy) | (Any & ~AlwaysFalsy) | ... omitted 3 union elements`
run_agent.py:13284: [invalid-argument-type] invalid-argument-type: Argument to function `_is_oauth_token` is incorrect: Expected `str`, found `str | dict[Unknown, Unknown] | Any | ... omitted 3 union elements`

✅ Fixed issues (3):

Rule	Count
`invalid-argument-type`	3

First entries

run_agent.py:13287: [invalid-argument-type] invalid-argument-type: Argument to function `len` is incorrect: Expected `Sized`, found `(str & ~AlwaysFalsy) | (dict[Unknown | str, Unknown | str | dict[str, str]] & ~AlwaysFalsy) | (Any & ~AlwaysFalsy) | ... omitted 3 union elements`
run_agent.py:7160: [invalid-argument-type] invalid-argument-type: Argument to function `build_anthropic_client` is incorrect: Expected `str`, found `str | dict[Unknown | str, Unknown | str | dict[str, str]] | Any | ... omitted 3 union elements`
run_agent.py:13284: [invalid-argument-type] invalid-argument-type: Argument to function `_is_oauth_token` is incorrect: Expected `str`, found `str | dict[Unknown | str, Unknown | str | dict[str, str]] | Any | ... omitted 3 union elements`

Unchanged: 4252 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@kjames2001

…uncation When edit_message_text exceeded Telegram's 4096 UTF-16 codepoint limit, the adapter caught the BadRequest, best-effort truncated the content with '…', and returned SendResult(success=True). The stream consumer believed the full edit was delivered and never recovered, silently dropping everything past the truncation boundary on long replies. Returning failure isn't safe either — the consumer's existing fallback path can race against the next streaming tick, producing duplicate sends or gaps. Instead, the adapter now SPLITS the oversized payload across the existing message + new continuation messages, so the user always gets the full reply in correct order. How it works: 1. Pre-flight: if utf16_len(content) already exceeds MAX_MESSAGE_LENGTH, call the new _edit_overflow_split helper directly — saves a doomed round-trip + a Telegram error. 2. Reactive: if Telegram still returns 'message_too_long' after the pre-flight (e.g. parse_mode formatting inflated the payload past the limit via MarkdownV2 escapes), the same helper handles it. 3. _edit_overflow_split: - Splits via truncate_message(len_fn=utf16_len) — same chunking the non-streaming send() path uses; chunks get '(1/N)' suffixes. - Edits the original message_id with chunk 1 (with parse_mode + plain-fallback when finalize=True, mirroring the main edit path). - Sends each remaining chunk via self._bot.send_message threaded as a reply to the previous chunk so the user sees them as a contiguous block. MarkdownV2-with-plain-fallback per chunk on finalize. - Returns SendResult(success=True, message_id=<last_chunk_id>, continuation_message_ids=(<chunk2_id>, <chunk3_id>, ...)) so the stream consumer can keep editing the most recent visible message and the gateway has full visibility into every message id. SendResult contract extension: Added optional continuation_message_ids: tuple = () field. When empty (the common case), behavior is unchanged. When populated, the caller knows the adapter delivered across multiple platform messages. Stream consumer integration: GatewayStreamConsumer._send_or_edit advances _message_id to the last-continuation id when it sees continuation_message_ids on a successful edit result, resets _last_sent_text (the new visible message holds only the final chunk's text), and fires on_new_message so tool-progress bubbles linearize below the new continuation rather than the original. Mirrors the openclaw #32535 inter-tool-leak guard. Composes with what just landed: - PR #23455 (UTF-16 length-aware splitting in stream consumer) prevents most overflows upstream by measuring text in UTF-16 codeunits before deciding to split. This PR is the safety net at the adapter boundary. - PR #23512 (native draft streaming, default for DM Telegram) routes DM streaming through send_draft, which has its own contract unaffected by this change. So this fix narrows in scope to the edit-based path: groups, supergroups, forum topics, every non-Telegram platform, and the per-response fallback after a draft failure. Salvage notes: - Cherry-picked from PR #19537 by @kjames2001. Original PR returned failure on overflow; this evolves to split-and-deliver so users never lose content and the consumer state stays consistent. - Dropped an unrelated model-picker hunk (line 2114-2117) that silently killed the 'X more available — type /model <name> directly' hint by hardcoding total=len(models). Not in scope. - Restored the timeout-aware retryable=not is_timeout signal in send()'s fallthrough catch block. Closes #19537.

@kjames2001

…uncation When edit_message_text exceeded Telegram's 4096 UTF-16 codepoint limit, the adapter caught the BadRequest, best-effort truncated the content with '…', and returned SendResult(success=True). The stream consumer believed the full edit was delivered and never recovered, silently dropping everything past the truncation boundary on long replies. Returning failure isn't safe either — the consumer's existing fallback path can race against the next streaming tick, producing duplicate sends or gaps. Instead, the adapter now SPLITS the oversized payload across the existing message + new continuation messages, so the user always gets the full reply in correct order. How it works: 1. Pre-flight: if utf16_len(content) already exceeds MAX_MESSAGE_LENGTH, call the new _edit_overflow_split helper directly — saves a doomed round-trip + a Telegram error. 2. Reactive: if Telegram still returns 'message_too_long' after the pre-flight (e.g. parse_mode formatting inflated the payload past the limit via MarkdownV2 escapes), the same helper handles it. 3. _edit_overflow_split: - Splits via truncate_message(len_fn=utf16_len) — same chunking the non-streaming send() path uses; chunks get '(1/N)' suffixes. - Edits the original message_id with chunk 1 (with parse_mode + plain-fallback when finalize=True, mirroring the main edit path). - Sends each remaining chunk via self._bot.send_message threaded as a reply to the previous chunk so the user sees them as a contiguous block. MarkdownV2-with-plain-fallback per chunk on finalize. - Returns SendResult(success=True, message_id=<last_chunk_id>, continuation_message_ids=(<chunk2_id>, <chunk3_id>, ...)) so the stream consumer can keep editing the most recent visible message and the gateway has full visibility into every message id. SendResult contract extension: Added optional continuation_message_ids: tuple = () field. When empty (the common case), behavior is unchanged. When populated, the caller knows the adapter delivered across multiple platform messages. Stream consumer integration: GatewayStreamConsumer._send_or_edit advances _message_id to the last-continuation id when it sees continuation_message_ids on a successful edit result, resets _last_sent_text (the new visible message holds only the final chunk's text), and fires on_new_message so tool-progress bubbles linearize below the new continuation rather than the original. Mirrors the openclaw #32535 inter-tool-leak guard. Composes with what just landed: - PR #23455 (UTF-16 length-aware splitting in stream consumer) prevents most overflows upstream by measuring text in UTF-16 codeunits before deciding to split. This PR is the safety net at the adapter boundary. - PR #23512 (native draft streaming, default for DM Telegram) routes DM streaming through send_draft, which has its own contract unaffected by this change. So this fix narrows in scope to the edit-based path: groups, supergroups, forum topics, every non-Telegram platform, and the per-response fallback after a draft failure. Salvage notes: - Cherry-picked from PR #19537 by @kjames2001. Original PR returned failure on overflow; this evolves to split-and-deliver so users never lose content and the consumer state stays consistent. - Dropped an unrelated model-picker hunk (line 2114-2117) that silently killed the 'X more available — type /model <name> directly' hint by hardcoding total=len(models). Not in scope. - Restored the timeout-aware retryable=not is_timeout signal in send()'s fallthrough catch block. Closes #19537.

@kjames2001

…uncation When edit_message_text exceeded Telegram's 4096 UTF-16 codepoint limit, the adapter caught the BadRequest, best-effort truncated the content with '…', and returned SendResult(success=True). The stream consumer believed the full edit was delivered and never recovered, silently dropping everything past the truncation boundary on long replies. Returning failure isn't safe either — the consumer's existing fallback path can race against the next streaming tick, producing duplicate sends or gaps. Instead, the adapter now SPLITS the oversized payload across the existing message + new continuation messages, so the user always gets the full reply in correct order. How it works: 1. Pre-flight: if utf16_len(content) already exceeds MAX_MESSAGE_LENGTH, call the new _edit_overflow_split helper directly — saves a doomed round-trip + a Telegram error. 2. Reactive: if Telegram still returns 'message_too_long' after the pre-flight (e.g. parse_mode formatting inflated the payload past the limit via MarkdownV2 escapes), the same helper handles it. 3. _edit_overflow_split: - Splits via truncate_message(len_fn=utf16_len) — same chunking the non-streaming send() path uses; chunks get '(1/N)' suffixes. - Edits the original message_id with chunk 1 (with parse_mode + plain-fallback when finalize=True, mirroring the main edit path). - Sends each remaining chunk via self._bot.send_message threaded as a reply to the previous chunk so the user sees them as a contiguous block. MarkdownV2-with-plain-fallback per chunk on finalize. - Returns SendResult(success=True, message_id=<last_chunk_id>, continuation_message_ids=(<chunk2_id>, <chunk3_id>, ...)) so the stream consumer can keep editing the most recent visible message and the gateway has full visibility into every message id. SendResult contract extension: Added optional continuation_message_ids: tuple = () field. When empty (the common case), behavior is unchanged. When populated, the caller knows the adapter delivered across multiple platform messages. Stream consumer integration: GatewayStreamConsumer._send_or_edit advances _message_id to the last-continuation id when it sees continuation_message_ids on a successful edit result, resets _last_sent_text (the new visible message holds only the final chunk's text), and fires on_new_message so tool-progress bubbles linearize below the new continuation rather than the original. Mirrors the openclaw NousResearch#32535 inter-tool-leak guard. Composes with what just landed: - PR NousResearch#23455 (UTF-16 length-aware splitting in stream consumer) prevents most overflows upstream by measuring text in UTF-16 codeunits before deciding to split. This PR is the safety net at the adapter boundary. - PR NousResearch#23512 (native draft streaming, default for DM Telegram) routes DM streaming through send_draft, which has its own contract unaffected by this change. So this fix narrows in scope to the edit-based path: groups, supergroups, forum topics, every non-Telegram platform, and the per-response fallback after a draft failure. Salvage notes: - Cherry-picked from PR NousResearch#19537 by @kjames2001. Original PR returned failure on overflow; this evolves to split-and-deliver so users never lose content and the consumer state stays consistent. - Dropped an unrelated model-picker hunk (line 2114-2117) that silently killed the 'X more available — type /model <name> directly' hint by hardcoding total=len(models). Not in scope. - Restored the timeout-aware retryable=not is_timeout signal in send()'s fallthrough catch block. Closes NousResearch#19537.

@kjames2001

…uncation When edit_message_text exceeded Telegram's 4096 UTF-16 codepoint limit, the adapter caught the BadRequest, best-effort truncated the content with '…', and returned SendResult(success=True). The stream consumer believed the full edit was delivered and never recovered, silently dropping everything past the truncation boundary on long replies. Returning failure isn't safe either — the consumer's existing fallback path can race against the next streaming tick, producing duplicate sends or gaps. Instead, the adapter now SPLITS the oversized payload across the existing message + new continuation messages, so the user always gets the full reply in correct order. How it works: 1. Pre-flight: if utf16_len(content) already exceeds MAX_MESSAGE_LENGTH, call the new _edit_overflow_split helper directly — saves a doomed round-trip + a Telegram error. 2. Reactive: if Telegram still returns 'message_too_long' after the pre-flight (e.g. parse_mode formatting inflated the payload past the limit via MarkdownV2 escapes), the same helper handles it. 3. _edit_overflow_split: - Splits via truncate_message(len_fn=utf16_len) — same chunking the non-streaming send() path uses; chunks get '(1/N)' suffixes. - Edits the original message_id with chunk 1 (with parse_mode + plain-fallback when finalize=True, mirroring the main edit path). - Sends each remaining chunk via self._bot.send_message threaded as a reply to the previous chunk so the user sees them as a contiguous block. MarkdownV2-with-plain-fallback per chunk on finalize. - Returns SendResult(success=True, message_id=<last_chunk_id>, continuation_message_ids=(<chunk2_id>, <chunk3_id>, ...)) so the stream consumer can keep editing the most recent visible message and the gateway has full visibility into every message id. SendResult contract extension: Added optional continuation_message_ids: tuple = () field. When empty (the common case), behavior is unchanged. When populated, the caller knows the adapter delivered across multiple platform messages. Stream consumer integration: GatewayStreamConsumer._send_or_edit advances _message_id to the last-continuation id when it sees continuation_message_ids on a successful edit result, resets _last_sent_text (the new visible message holds only the final chunk's text), and fires on_new_message so tool-progress bubbles linearize below the new continuation rather than the original. Mirrors the openclaw NousResearch#32535 inter-tool-leak guard. Composes with what just landed: - PR NousResearch#23455 (UTF-16 length-aware splitting in stream consumer) prevents most overflows upstream by measuring text in UTF-16 codeunits before deciding to split. This PR is the safety net at the adapter boundary. - PR NousResearch#23512 (native draft streaming, default for DM Telegram) routes DM streaming through send_draft, which has its own contract unaffected by this change. So this fix narrows in scope to the edit-based path: groups, supergroups, forum topics, every non-Telegram platform, and the per-response fallback after a draft failure. Salvage notes: - Cherry-picked from PR NousResearch#19537 by @kjames2001. Original PR returned failure on overflow; this evolves to split-and-deliver so users never lose content and the consumer state stays consistent. - Dropped an unrelated model-picker hunk (line 2114-2117) that silently killed the 'X more available — type /model <name> directly' hint by hardcoding total=len(models). Not in scope. - Restored the timeout-aware retryable=not is_timeout signal in send()'s fallthrough catch block. Closes NousResearch#19537.

@kjames2001

…uncation When edit_message_text exceeded Telegram's 4096 UTF-16 codepoint limit, the adapter caught the BadRequest, best-effort truncated the content with '…', and returned SendResult(success=True). The stream consumer believed the full edit was delivered and never recovered, silently dropping everything past the truncation boundary on long replies. Returning failure isn't safe either — the consumer's existing fallback path can race against the next streaming tick, producing duplicate sends or gaps. Instead, the adapter now SPLITS the oversized payload across the existing message + new continuation messages, so the user always gets the full reply in correct order. How it works: 1. Pre-flight: if utf16_len(content) already exceeds MAX_MESSAGE_LENGTH, call the new _edit_overflow_split helper directly — saves a doomed round-trip + a Telegram error. 2. Reactive: if Telegram still returns 'message_too_long' after the pre-flight (e.g. parse_mode formatting inflated the payload past the limit via MarkdownV2 escapes), the same helper handles it. 3. _edit_overflow_split: - Splits via truncate_message(len_fn=utf16_len) — same chunking the non-streaming send() path uses; chunks get '(1/N)' suffixes. - Edits the original message_id with chunk 1 (with parse_mode + plain-fallback when finalize=True, mirroring the main edit path). - Sends each remaining chunk via self._bot.send_message threaded as a reply to the previous chunk so the user sees them as a contiguous block. MarkdownV2-with-plain-fallback per chunk on finalize. - Returns SendResult(success=True, message_id=<last_chunk_id>, continuation_message_ids=(<chunk2_id>, <chunk3_id>, ...)) so the stream consumer can keep editing the most recent visible message and the gateway has full visibility into every message id. SendResult contract extension: Added optional continuation_message_ids: tuple = () field. When empty (the common case), behavior is unchanged. When populated, the caller knows the adapter delivered across multiple platform messages. Stream consumer integration: GatewayStreamConsumer._send_or_edit advances _message_id to the last-continuation id when it sees continuation_message_ids on a successful edit result, resets _last_sent_text (the new visible message holds only the final chunk's text), and fires on_new_message so tool-progress bubbles linearize below the new continuation rather than the original. Mirrors the openclaw NousResearch#32535 inter-tool-leak guard. Composes with what just landed: - PR NousResearch#23455 (UTF-16 length-aware splitting in stream consumer) prevents most overflows upstream by measuring text in UTF-16 codeunits before deciding to split. This PR is the safety net at the adapter boundary. - PR NousResearch#23512 (native draft streaming, default for DM Telegram) routes DM streaming through send_draft, which has its own contract unaffected by this change. So this fix narrows in scope to the edit-based path: groups, supergroups, forum topics, every non-Telegram platform, and the per-response fallback after a draft failure. Salvage notes: - Cherry-picked from PR NousResearch#19537 by @kjames2001. Original PR returned failure on overflow; this evolves to split-and-deliver so users never lose content and the consumer state stays consistent. - Dropped an unrelated model-picker hunk (line 2114-2117) that silently killed the 'X more available — type /model <name> directly' hint by hardcoding total=len(models). Not in scope. - Restored the timeout-aware retryable=not is_timeout signal in send()'s fallthrough catch block. Closes NousResearch#19537.

@kjames2001

…uncation When edit_message_text exceeded Telegram's 4096 UTF-16 codepoint limit, the adapter caught the BadRequest, best-effort truncated the content with '…', and returned SendResult(success=True). The stream consumer believed the full edit was delivered and never recovered, silently dropping everything past the truncation boundary on long replies. Returning failure isn't safe either — the consumer's existing fallback path can race against the next streaming tick, producing duplicate sends or gaps. Instead, the adapter now SPLITS the oversized payload across the existing message + new continuation messages, so the user always gets the full reply in correct order. How it works: 1. Pre-flight: if utf16_len(content) already exceeds MAX_MESSAGE_LENGTH, call the new _edit_overflow_split helper directly — saves a doomed round-trip + a Telegram error. 2. Reactive: if Telegram still returns 'message_too_long' after the pre-flight (e.g. parse_mode formatting inflated the payload past the limit via MarkdownV2 escapes), the same helper handles it. 3. _edit_overflow_split: - Splits via truncate_message(len_fn=utf16_len) — same chunking the non-streaming send() path uses; chunks get '(1/N)' suffixes. - Edits the original message_id with chunk 1 (with parse_mode + plain-fallback when finalize=True, mirroring the main edit path). - Sends each remaining chunk via self._bot.send_message threaded as a reply to the previous chunk so the user sees them as a contiguous block. MarkdownV2-with-plain-fallback per chunk on finalize. - Returns SendResult(success=True, message_id=<last_chunk_id>, continuation_message_ids=(<chunk2_id>, <chunk3_id>, ...)) so the stream consumer can keep editing the most recent visible message and the gateway has full visibility into every message id. SendResult contract extension: Added optional continuation_message_ids: tuple = () field. When empty (the common case), behavior is unchanged. When populated, the caller knows the adapter delivered across multiple platform messages. Stream consumer integration: GatewayStreamConsumer._send_or_edit advances _message_id to the last-continuation id when it sees continuation_message_ids on a successful edit result, resets _last_sent_text (the new visible message holds only the final chunk's text), and fires on_new_message so tool-progress bubbles linearize below the new continuation rather than the original. Mirrors the openclaw NousResearch#32535 inter-tool-leak guard. Composes with what just landed: - PR NousResearch#23455 (UTF-16 length-aware splitting in stream consumer) prevents most overflows upstream by measuring text in UTF-16 codeunits before deciding to split. This PR is the safety net at the adapter boundary. - PR NousResearch#23512 (native draft streaming, default for DM Telegram) routes DM streaming through send_draft, which has its own contract unaffected by this change. So this fix narrows in scope to the edit-based path: groups, supergroups, forum topics, every non-Telegram platform, and the per-response fallback after a draft failure. Salvage notes: - Cherry-picked from PR NousResearch#19537 by @kjames2001. Original PR returned failure on overflow; this evolves to split-and-deliver so users never lose content and the consumer state stays consistent. - Dropped an unrelated model-picker hunk (line 2114-2117) that silently killed the 'X more available — type /model <name> directly' hint by hardcoding total=len(models). Not in scope. - Restored the timeout-aware retryable=not is_timeout signal in send()'s fallthrough catch block. Closes NousResearch#19537.

@kjames2001

…uncation When edit_message_text exceeded Telegram's 4096 UTF-16 codepoint limit, the adapter caught the BadRequest, best-effort truncated the content with '…', and returned SendResult(success=True). The stream consumer believed the full edit was delivered and never recovered, silently dropping everything past the truncation boundary on long replies. Returning failure isn't safe either — the consumer's existing fallback path can race against the next streaming tick, producing duplicate sends or gaps. Instead, the adapter now SPLITS the oversized payload across the existing message + new continuation messages, so the user always gets the full reply in correct order. How it works: 1. Pre-flight: if utf16_len(content) already exceeds MAX_MESSAGE_LENGTH, call the new _edit_overflow_split helper directly — saves a doomed round-trip + a Telegram error. 2. Reactive: if Telegram still returns 'message_too_long' after the pre-flight (e.g. parse_mode formatting inflated the payload past the limit via MarkdownV2 escapes), the same helper handles it. 3. _edit_overflow_split: - Splits via truncate_message(len_fn=utf16_len) — same chunking the non-streaming send() path uses; chunks get '(1/N)' suffixes. - Edits the original message_id with chunk 1 (with parse_mode + plain-fallback when finalize=True, mirroring the main edit path). - Sends each remaining chunk via self._bot.send_message threaded as a reply to the previous chunk so the user sees them as a contiguous block. MarkdownV2-with-plain-fallback per chunk on finalize. - Returns SendResult(success=True, message_id=<last_chunk_id>, continuation_message_ids=(<chunk2_id>, <chunk3_id>, ...)) so the stream consumer can keep editing the most recent visible message and the gateway has full visibility into every message id. SendResult contract extension: Added optional continuation_message_ids: tuple = () field. When empty (the common case), behavior is unchanged. When populated, the caller knows the adapter delivered across multiple platform messages. Stream consumer integration: GatewayStreamConsumer._send_or_edit advances _message_id to the last-continuation id when it sees continuation_message_ids on a successful edit result, resets _last_sent_text (the new visible message holds only the final chunk's text), and fires on_new_message so tool-progress bubbles linearize below the new continuation rather than the original. Mirrors the openclaw NousResearch#32535 inter-tool-leak guard. Composes with what just landed: - PR NousResearch#23455 (UTF-16 length-aware splitting in stream consumer) prevents most overflows upstream by measuring text in UTF-16 codeunits before deciding to split. This PR is the safety net at the adapter boundary. - PR NousResearch#23512 (native draft streaming, default for DM Telegram) routes DM streaming through send_draft, which has its own contract unaffected by this change. So this fix narrows in scope to the edit-based path: groups, supergroups, forum topics, every non-Telegram platform, and the per-response fallback after a draft failure. Salvage notes: - Cherry-picked from PR NousResearch#19537 by @kjames2001. Original PR returned failure on overflow; this evolves to split-and-deliver so users never lose content and the consumer state stays consistent. - Dropped an unrelated model-picker hunk (line 2114-2117) that silently killed the 'X more available — type /model <name> directly' hint by hardcoding total=len(models). Not in scope. - Restored the timeout-aware retryable=not is_timeout signal in send()'s fallthrough catch block. Closes NousResearch#19537.

@kjames2001

…uncation When edit_message_text exceeded Telegram's 4096 UTF-16 codepoint limit, the adapter caught the BadRequest, best-effort truncated the content with '…', and returned SendResult(success=True). The stream consumer believed the full edit was delivered and never recovered, silently dropping everything past the truncation boundary on long replies. Returning failure isn't safe either — the consumer's existing fallback path can race against the next streaming tick, producing duplicate sends or gaps. Instead, the adapter now SPLITS the oversized payload across the existing message + new continuation messages, so the user always gets the full reply in correct order. How it works: 1. Pre-flight: if utf16_len(content) already exceeds MAX_MESSAGE_LENGTH, call the new _edit_overflow_split helper directly — saves a doomed round-trip + a Telegram error. 2. Reactive: if Telegram still returns 'message_too_long' after the pre-flight (e.g. parse_mode formatting inflated the payload past the limit via MarkdownV2 escapes), the same helper handles it. 3. _edit_overflow_split: - Splits via truncate_message(len_fn=utf16_len) — same chunking the non-streaming send() path uses; chunks get '(1/N)' suffixes. - Edits the original message_id with chunk 1 (with parse_mode + plain-fallback when finalize=True, mirroring the main edit path). - Sends each remaining chunk via self._bot.send_message threaded as a reply to the previous chunk so the user sees them as a contiguous block. MarkdownV2-with-plain-fallback per chunk on finalize. - Returns SendResult(success=True, message_id=<last_chunk_id>, continuation_message_ids=(<chunk2_id>, <chunk3_id>, ...)) so the stream consumer can keep editing the most recent visible message and the gateway has full visibility into every message id. SendResult contract extension: Added optional continuation_message_ids: tuple = () field. When empty (the common case), behavior is unchanged. When populated, the caller knows the adapter delivered across multiple platform messages. Stream consumer integration: GatewayStreamConsumer._send_or_edit advances _message_id to the last-continuation id when it sees continuation_message_ids on a successful edit result, resets _last_sent_text (the new visible message holds only the final chunk's text), and fires on_new_message so tool-progress bubbles linearize below the new continuation rather than the original. Mirrors the openclaw NousResearch#32535 inter-tool-leak guard. Composes with what just landed: - PR NousResearch#23455 (UTF-16 length-aware splitting in stream consumer) prevents most overflows upstream by measuring text in UTF-16 codeunits before deciding to split. This PR is the safety net at the adapter boundary. - PR NousResearch#23512 (native draft streaming, default for DM Telegram) routes DM streaming through send_draft, which has its own contract unaffected by this change. So this fix narrows in scope to the edit-based path: groups, supergroups, forum topics, every non-Telegram platform, and the per-response fallback after a draft failure. Salvage notes: - Cherry-picked from PR NousResearch#19537 by @kjames2001. Original PR returned failure on overflow; this evolves to split-and-deliver so users never lose content and the consumer state stays consistent. - Dropped an unrelated model-picker hunk (line 2114-2117) that silently killed the 'X more available — type /model <name> directly' hint by hardcoding total=len(models). Not in scope. - Restored the timeout-aware retryable=not is_timeout signal in send()'s fallthrough catch block. Closes NousResearch#19537.

freemanconsulting and others added 3 commits May 10, 2026 16:15

chore: AUTHOR_MAP entry for Freeman-Consulting

62065f0

teknium1 merged commit c0bbdec into main May 10, 2026
13 of 16 checks passed

teknium1 deleted the salvage/pr-11170 branch May 10, 2026 23:21

teknium1 mentioned this pull request May 10, 2026

fix: use UTF-16 length for Telegram stream consumer message splitting #11170

Closed

This was referenced May 11, 2026

Ecosystem Digest — 2026-05-11 Haderach-Ram/openclaw-radar#2

Open

Ecosystem Digest — 2026-05-11 Haderach-Ram/openclaw-radar#3

Closed

teknium1 mentioned this pull request May 11, 2026

fix(telegram): split-and-deliver oversized edits instead of silent truncation (salvage of #19537) #23576

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(stream-consumer): use UTF-16 length for Telegram message splitting (salvage of #11170)#23455

fix(stream-consumer): use UTF-16 length for Telegram message splitting (salvage of #11170)#23455
teknium1 merged 3 commits into
mainfrom
salvage/pr-11170

teknium1 commented May 10, 2026

Uh oh!

github-actions Bot commented May 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

teknium1 commented May 10, 2026

Summary

Changes

Improvements during salvage

Validation

Interaction with PR #19537

Uh oh!

github-actions Bot commented May 10, 2026

🔎 Lint report: salvage/pr-11170 vs origin/main

ruff

ty (type checker)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

🔎 Lint report: `salvage/pr-11170` vs `origin/main`