Skip to content

Add Telegram Bot API 10.1 rich message support#44780

Closed
ITheEqualizer wants to merge 2 commits into
NousResearch:mainfrom
ITheEqualizer:feat/telegram-rich-messages
Closed

Add Telegram Bot API 10.1 rich message support#44780
ITheEqualizer wants to merge 2 commits into
NousResearch:mainfrom
ITheEqualizer:feat/telegram-rich-messages

Conversation

@ITheEqualizer

Copy link
Copy Markdown
Contributor

Summary

Telegram's Bot API 10.1 (June 11 2026) added Rich Messages (sendRichMessage / sendRichMessageDraft / InputRichMessage). This wires the Telegram gateway to use them so replies render tables, task lists, headings, nested blockquotes, collapsible <details>, footnotes/references, math/formulas, underline, sub/superscript, marked text, and anchors natively, instead of flattening tables into row-group bullets.

  • Final replies now send the agent's raw markdown via sendRichMessage.
  • DM streaming previews now use sendRichMessageDraft, so the animated draft matches the final rich message.
  • The agent prompt hint is updated to encourage rich Markdown (it previously said "Telegram has NO table syntax").
  • On by default; opt out with platforms.telegram.extra.rich_messages: false.

python-telegram-bot 22.6 doesn't expose a typed send_rich_message, so we call the raw endpoint via Bot.do_api_request("sendRichMessage"/"sendRichMessageDraft", api_kwargs=..., return_type=Message).

Motivation

The legacy path runs every reply through MarkdownV2 (format_message), which escapes/destroys rich syntax — pipe tables became "row-group bullets" and the prompt hint actively told agents to avoid tables. With Rich Messages, the raw markdown renders server-side, so structured output (tables, task lists) finally works on Telegram.

What changed

gateway/platforms/telegram.py

  • __init__: rich_messages opt-out flag (default on) + a latch for draft capability failures.
  • New helpers: rich length pre-check (32,768 UTF-8 bytes), async-capability gate (do_api_request is a coroutine fn), raw-markdown payload builder, conservative fallback-error classifier, single-send routing mirror, and _try_send_rich / _try_send_rich_draft.
  • send(): rich fast-path before the MarkdownV2 path; falls through to legacy on permanent/capability errors, returns directly on success or transient failure.
  • send_draft(): tries sendRichMessageDraft, with the existing plain-text sendMessageDraft as the fallback frame.

agent/prompt_builder.py — rewrote PLATFORM_HINTS["telegram"] to encourage rich Markdown; kept the include MEDIA:/absolute/path guidance. (Static stable-tier text → prompt-cache safe.)

Docs/config — documented rich_messages in cli-config.yaml.example; rewrote the rendering section in website/docs/user-guide/messaging/telegram.md (+ zh-Hans i18n copy).

Tests — new tests/gateway/test_telegram_rich_messages.py (21 tests) + a prompt-hint regression in tests/agent/test_prompt_builder.py.

Safety / design

  • Capability gate uses inspect.iscoroutinefunction(bot.do_api_request) — real bots + opt-in mocks engage rich; plain mocks fall through to legacy unchanged.
  • Fallback only on permanent errors (BadRequest / capability / unknown-endpoint / 404 / "method not found") → legacy MarkdownV2. Transient/network/timeout errors do NOT legacy-resend the final message (the rich request may have landed — avoids duplicates); they return a failure with the legacy retry semantics.
  • Routing parity reuses _thread_kwargs_for_send / _is_private_dm_topic_send: preserves message_thread_id, direct_messages_topic_id (dropping the paired None thread id), reply anchor (scalar reply_to_message_id), disable_notification, and the DM-topic fail-loud refusal (delegated to legacy as the single source).
  • Limits: content over 32,768 UTF-8 bytes skips rich and uses legacy chunking; block/column overflow degrades via BadRequest fallback.
  • Streaming drafts are ephemeral, so a duplicate/lost rich draft is harmless — any failure renders the legacy plain-text frame; a capability failure latches off subsequent rich-draft attempts. Edit-based streaming (groups) and the mid-stream edit path stay MarkdownV2 (Telegram has no rich-edit method).
  • Out of scope (deliberate): the structured InputRichMessage block model (RichBlock*/RichText*), the html input variant, and rich message editing.

Config

On by default. To disable per platform:

gateway:
  platforms:
    telegram:
      extra:
        rich_messages: false

Tests

Run with -o 'addopts=':

  • tests/gateway/test_telegram_rich_messages.py21 passed (16 final-send + 5 streaming-draft).
  • Targeted: test_telegram_rich_messages.py + test_stream_consumer_draft.py + tests/agent/test_prompt_builder.py157 passed, 1 skipped.
  • Per-file regression (all green): telegram_format 101, thread_fallback 43, topic_mode 43, reply_mode 40, reply_quote 4, stream_consumer 89, stream_consumer_draft 12, stream_consumer_fresh_final 15, stream_consumer_thread_routing 7, dm_topics 36.
  • Full tests/gateway (single process): base-vs-branch failure set is identical — the only differences are the new rich tests now passing. The 13 remaining failures (simplex ModuleNotFoundError, wecom, shutdown_forensics, and the approval/model_picker/slash_confirm files that inject their own fake telegram module) are a pre-existing cross-file sys.modules pollution artifact that reproduces on main and is unrelated to this change.

Deploy

Requires a gateway restart (/restart or hermes gateway restart).

Caveat

Validated against mocked unit tests, not a live bot. Since the endpoint shipped ~1 day ago, if the exact rich_message payload shape differs in practice, the graceful fallback keeps existing sends working — rich rendering just won't engage until confirmed live.

Introduce opportunistic support for Telegram Bot API 10.1 rich messages by sending raw agent Markdown via sendRichMessage and streaming previews via sendRichMessageDraft. Implements a rich-path fast‑path in gateway/platforms/telegram.py (RICH_MESSAGE_MAX_BYTES=32768, feature gate platforms.telegram.extra.rich_messages, bot capability checks, routing/thread handling, and conservative fallback rules: permanent/capability errors fall back to the legacy MarkdownV2 path, transient/network errors are surfaced without legacy-resend). Also add a latch for draft capability failures (_rich_draft_disabled) and preserve legacy chunking and draft behavior when needed. Update agent prompt hints (telegram encourages rich Markdown/tables), add CLI config example option, update English and Chinese docs to describe rich messages and fallbacks, and add/adjust tests for rich send and draft behavior.
@alt-glitch alt-glitch added type/feature New feature or request comp/gateway Gateway runner, session dispatch, delivery platform/telegram Telegram bot adapter P3 Low — cosmetic, nice to have labels Jun 12, 2026
AIalliAI pushed a commit to AIalliAI/Hermes that referenced this pull request Jun 12, 2026
…drafts)

Cherry-pick of upstream PR NousResearch#44780 (ITheEqualizer). Final replies send raw
agent markdown via sendRichMessage; DM streaming previews use
sendRichMessageDraft; MarkdownV2 stays as the fallback path. Opt out with
platforms.telegram.extra.rich_messages: false.

Resolved a trivial conflict with main's supports_code_blocks attribute
(kept both).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AIalliAI

Copy link
Copy Markdown
Contributor

Verified this end-to-end — the post-cutoff API claims are real and the tests are genuine. Details below, plus one small bug and a few notes.

API verification (live docs, since 10.1 shipped a day ago):

  • sendRichMessage, sendRichMessageDraft, and InputRichMessage all exist as described; the {"markdown": ...} payload shape and the draft params (chat_id/draft_id/rich_message/message_thread_id) match the reference exactly.
  • Negative control: reverting telegram.py + prompt_builder.py to main while keeping the PR's tests → 16/21 rich tests fail (the 5 survivors are the assert-legacy-behavior ones). The tests exercise the new path for real.
  • Regression sweep (format, thread_fallback, topic_mode, reply_mode, reply_quote, send_draft_format, send_path_health, stream_consumer*) green. The thread_fallback failures that show up when several telegram files run in one process reproduce identically on clean main — pre-existing cross-file order interference, not this change.
  • The _compute_single_send_routing mirror is faithful to send()'s index-0 block, including the dm_topic_reply_to_off carve-out and the fail-loud delegation.

Bug (small): pool timeouts become non-retryable on the rich path. Legacy send()'s outer handler returns retryable=(is_connect_timeout or is_pool_timeout or not is_timeout) — with an explicit comment that an httpx pool timeout means the request was never sent and must not be silently dropped. _try_send_rich's transient branch computes retryable=(is_connect_timeout or not is_timeout), so a pool-exhausted rich send returns retryable=False and the message is silently lost. Suggest adding self._looks_like_pool_timeout(exc) to the disjunction.

Note: reply_to_message_id is undocumented on sendRichMessage. The documented reply param is reply_parameters; the legacy scalar was never documented for post-7.0 methods. It does work today — I checked the official server source (tdlib/telegram-bot-api Client.cpp): do_send_message parses replies via get_reply_parameters(query), which falls back to the legacy reply_to_message_id arg generically for every send method. So no functional issue against the official server, but it's an undocumented contract; reply_parameters: {"message_id": N} would be the spec-clean spelling and serializes through api_kwargs just as well.

Note: the "Telegram exposes no rich-edit method" comment is inaccurate. 10.1 added a rich_message parameter to editMessageText (it's in the changelog). Keeping edit-based streaming on MarkdownV2 is a perfectly reasonable scope cut, but the code comment / PR rationale should say "deferred" rather than "impossible" — and it means group (edit-based) streaming is upgradeable in a follow-up.

Note: extra.disable_link_previews doesn't apply on the rich path. sendRichMessage exposes no link_preview_options parameter, so users who set that flag will see previews come back whenever the rich path engages. Probably worth a sentence in the docs section (or passing skip_entity_detection when the flag is set, if that suppresses URL entity previews — untested).

Nit: _rich_message_payload's skip_entity_detection parameter is currently dead — no caller passes it.

Also: this implements #44428 (where the blocker discussed was the PTB 22.6 pin — the raw do_api_request route neatly sidesteps that); consider adding Fixes #44428 to the body so they link. The branch currently conflicts with main, but it's one trivial hunk (supports_code_blocks landed where RICH_MESSAGE_MAX_BYTES is inserted — keep both).

Cherry-picked onto our rollup branch (with that conflict resolved) and everything above ran against the live venv. Nice work on the fallback taxonomy — the no-resend-on-transient rule for finals vs. the latch-off for drafts is exactly the right asymmetry.

@AIalliAI

Copy link
Copy Markdown
Contributor

Heads-up: this went CONFLICTING after #41215 (dde9c0d19) landed on main — it added a supports_code_blocks = True class attribute to TelegramAdapter at the same spot where this PR adds RICH_MESSAGE_MAX_BYTES. It's the only conflict (single hunk in gateway/platforms/telegram.py); the other six files apply cleanly.

Resolution is keep-both:

    # Telegram message limits
    MAX_MESSAGE_LENGTH = 4096
    supports_code_blocks = True  # Telegram MarkdownV2 renders fenced code blocks
    # Bot API 10.1 Rich Messages cap the raw markdown/html text at 32,768
    # UTF-8 bytes. Content above this is sent via the legacy chunking path.
    RICH_MESSAGE_MAX_BYTES = 32768

Verified the merged result compiles with both attributes present — the two changes don't interact (supports_code_blocks is read by the markdown rendering path, not the rich-message send path).

Improve Telegram adapter rich-message support: switch the rich limit to a 32,768-character cap, add robust handling for missing-endpoint/capability errors (latch off rich sends), separate capability vs content BadRequest classification, and implement retry/flood-control/backoff logic for sendRichMessage. Add _rich_send_disabled flag and honor metadata.expect_edits to keep messages that will be edited on the legacy (editable) MarkdownV2 path. Carry reply parameters as reply_parameters, expose link_preview options, and ensure drafts only latch off on true capability failures (not per-frame parse errors). Propagate expect_edits from stream consumers and runner progress code so preview/status bubbles remain editable. Update tests to cover new behaviors and adjust docs (including i18n) to note the character limit, latching behavior, and flood-control semantics.
AIalliAI added a commit to AIalliAI/Hermes that referenced this pull request Jun 12, 2026
…d path

Parity with the legacy send() loop (dc4de14 / NousResearch#35664): an httpx pool
timeout explicitly never left the process, so the rich path now retries
it in place and surfaces retryable=True instead of silently dropping the
message. Flagged in the NousResearch#44780 review; the PR branch predates the legacy
pool-timeout fix, so it could not mirror it.
@AIalliAI

Copy link
Copy Markdown
Contributor

Verified the follow-up commit (aa9c890) the same way as the original — negative control plus full suite against a current-main-merged branch. Summary: it resolves nearly everything raised here and in #44829's follow-ups, two items remain.

Verification:

  • Negative control: the updated tests against the v1 implementation → 18/39 rich tests + 5 progress/stream-routing tests fail; with the new implementation all pass (39 + 33 + 7).
  • Regression sweep (format, thread_fallback, topic_mode, reply_mode, reply_quote, send_draft_format, send_path_health, network, progress_edit_transient, stream_consumer×3): 440 tests, all green.
  • Live-spec checks: the 32,768 limit is indeed characters (the bytes→chars fix is correct); reply_parameters is the documented reply param ✓; the PTB error shapes in _is_rich_capability_error (EndPointNotFound on 404, InvalidToken wrap) match PTB's real behavior — nice catch, my earlier suggestion of plain message-matching would have missed the InvalidToken wrap.

Confirmed fixed from the earlier review / #44829: reply_parameters ✓, send-side capability latch ✓ (and the draft latch narrowed so mid-stream parse 400s don't latch — an improvement over both earlier versions, since draft frames carry partial markdown), expect_edits threading ✓ (this is the right fix for the format-flip-on-first-edit hazard, and it's new relative to #44829), dropping return_type=Message ✓ (a post-delivery parse failure can no longer masquerade as a send failure — the negative control actually tripped over the old msg.get("result", {}) crash on None), flood-control + connect/network retry parity ✓.

Still open (1): pool timeouts remain non-retryable on the rich path. The final transient return is still retryable=(is_connect_timeout or not is_timeout). Root cause: this branch is based on a main snapshot that predates dc4de14 (#35664), so _looks_like_pool_timeout doesn't exist here and the branch's legacy path lacks the pool handling too — merging won't regress legacy (main keeps its version), but the rich path needs the mirror. After a rebase onto current main it's two lines: add is_pool_timeout = self._looks_like_pool_timeout(exc) next to the other classifiers, include it in the in-loop retry disjunction (a pool timeout is explicitly "not sent to Telegram", so in-place retry is duplicate-safe) and in the final retryable=. A rebase would also clear the #41215 supports_code_blocks conflict.

Still open (2): link_preview_options is not a sendRichMessage parameter. Checked the live method spec — its full parameter list is business_connection_id, chat_id, message_thread_id, direct_messages_topic_id, rich_message, disable_notification, protect_content, allow_paid_broadcast, message_effect_id, suggested_post_parameters, reply_parameters, reply_markup — and InputRichMessage only has html|markdown, is_rtl, skip_entity_detection. Unknown params are silently ignored, so the new payload["link_preview_options"] line is a no-op and extra.disable_link_previews users still get previews on the rich path. Options: drop the line and document the limitation, or pass skip_entity_detection: true when the flag is set (suppresses URL entity detection, which may be the closest spec-level equivalent — behavior worth a live test).

Cherry-picked onto our rollup branch (same single keep-both conflict, plus the two-line pool-timeout mirror since that branch has _looks_like_pool_timeout) — CI runs there will exercise it. Also: since #44829 carries v1 + its own follow-ups, the cleanest consolidation path may be rebasing this branch onto main and letting #44829 cherry-pick the result, as you suggested there.

@teknium1

Copy link
Copy Markdown
Contributor

Merged via PR #44829 — your commit was cherry-picked onto current main with your authorship preserved in git history (05b9c84), via rebase-merge. Excellent work: the endpoint shipped a day before you wired it, and the fallback design (permanent-vs-transient classification, no duplicate resends, routing parity) was careful and correct.

Three follow-ups were added on top during salvage:

  1. Reply anchoring now uses reply_parameters per the sendRichMessage spec — the reply_to_message_id scalar isn't documented on that endpoint, and the Bot API silently ignores unknown params, so replies would have quietly lost their anchor.
  2. Final sends now latch rich off after an endpoint-capability failure (mirroring your draft latch), so old-PTB setups don't pay a failed roundtrip on every reply.
  3. rich_messages ships default OFF for now — the endpoint is days old and we want live validation against a real bot before flipping the default on (your PR honestly flagged the same caveat). Flipping it to on-by-default is the planned follow-up once validated.

Thanks for the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have platform/telegram Telegram bot adapter type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants