feat(desktop+gateway): remote media relay — attach images/PDFs & display gateway images over the network#41336
Merged
Merged
Conversation
…splay gateway images over the network
Desktop connected to a remote gateway can now attach images and PDFs and
display agent-written images. Previously the desktop passed a LOCAL file path
to image.attach; on a remote gateway that path doesn't exist, so the image was
silently dropped ("skipped unreadable path") and the vision model never saw it.
The reverse direction was also broken — images the agent wrote on the gateway
rendered as dead links in the remote client.
Gateway (tui_gateway/server.py):
- image.attach_bytes: base64 byte upload written into the gateway's own images
dir and queued via the existing native-image-attach pipeline. Magic-byte
extension sniffing, data-URL prefix + whitespace tolerance, 25 MB cap,
structured error codes. Accepts content_base64/filename (canonical) and
data/ext (older-desktop aliases).
- pdf.attach: renders each page to PNG via pdftoppm (poppler-utils) at 150 DPI
and queues the pages as images; 50 MB / 25-page caps. Accepts host path or
base64 upload.
- Shared helpers (_decode_attach_base64, _sniff_image_ext, _queue_attached_image)
so the two methods and the existing image.attach don't duplicate logic.
Gateway (hermes_cli/web_server.py):
- GET /api/media: returns a gateway-local image as a base64 data URL so remote
clients can display it. Auth-gated like every /api route, extension
allowlist + size cap, AND confined to the gateway's own media roots
(images/screenshots/cache, resolved symlink-safe) so an authed caller can't
read image-extension files anywhere on disk.
Desktop (apps/desktop):
- syncImageAttachmentsForSubmit uploads bytes via image.attach_bytes when the
connection mode is 'remote'; the local fast path is unchanged.
- media.ts gains isRemoteGateway() + gatewayMediaDataUrl(); directive-text and
markdown-text fetch images over /api/media in remote mode.
Consolidates the competing remote-media PRs (#38876, #40317, #21908, #39437)
into one coherent implementation, taking the strongest parts of each and adding
shared-helper cleanup plus the /api/media root-confinement hardening on top.
The per-profile gateway switching from #38876 is intentionally left out as a
separable feature. TUI file uploads (#40492) remain a separate surface.
Tested: 11 new tui_gateway tests + 5 /api/media endpoint tests + desktop
media.remote unit tests; full tui_gateway + web_server suites green (472
passed); tsc -b clean; E2E verified the full attach→disk→queue and
gateway-path→data-URL display round-trip plus the out-of-root security block.
Co-authored-by: Max Mitcham <maxmitcham@mac.home>
Co-authored-by: Justlrnal4 <Justlrnal4@users.noreply.github.com>
Co-authored-by: Chris Cook <ccook@nvms.com>
Co-authored-by: Thomas Paquette <thomas.paquette@gmail.com>
Contributor
🔎 Lint report:
|
This was referenced Jun 7, 2026
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Desktop connected to a remote gateway can now attach images and PDFs and display agent-written images — the whole remote-media round-trip works for the first time.
Previously the desktop passed a local file path to
image.attach. On a remote gateway (VPS) that path doesn't exist, so the upload was silently dropped ("skipped unreadable path") and the vision model never saw the image. The reverse direction was broken too: images the agent wrote on the gateway rendered as dead links in the remote client.This consolidates the competing remote-media PRs (#38876, #40317, #21908, #39437) into one coherent implementation, taking the strongest part of each and adding shared-helper cleanup + a security hardening of
/api/mediaon top. Salvaged onto currentmain(PR #41203 went stale by 10 commits during review).Changes
Gateway —
tui_gateway/server.pyimage.attach_bytes: base64 byte upload, written into the gateway's ownimages/dir and queued via the existing native-image-attach pipeline. Magic-byte extension sniffing, data-URL prefix + whitespace tolerance, 25 MB cap, structured error codes. Acceptscontent_base64/filename(canonical) anddata/ext(older-desktop aliases).pdf.attach: renders each page to PNG viapdftoppm(poppler-utils) at 150 DPI and queues the pages as images; 50 MB / 25-page caps. Accepts hostpathor base64 upload._decode_attach_base64,_sniff_image_ext,_queue_attached_image) so the two new methods and the existingimage.attachdon't duplicate logic.Gateway —
hermes_cli/web_server.pyGET /api/media: returns a gateway-local image as a base64 data URL so remote clients can display it. Auth-gated like every/apiroute, extension allowlist + size cap, and confined to the gateway's own media roots (images/screenshots/cache, resolved symlink-safe) so an authed caller can't read image-extension files anywhere on disk.Desktop —
apps/desktopsyncImageAttachmentsForSubmituploads bytes viaimage.attach_byteswhen the connection mode is'remote'; the local fast path is unchanged.media.tsgainsisRemoteGateway()+gatewayMediaDataUrl();directive-textandmarkdown-textfetch images over/api/mediain remote mode.scripts/release.py— AUTHOR_MAP entries for the salvaged contributors.What's intentionally out of scope
Validation
/api/mediaas data URL/api/mediascopetui_gatewaytests + 5/api/mediaendpoint tests + desktopmedia.remoteunit tests.web_server(uvicorn) against an isolatedHERMES_HOME, droveimage.attach_bytes+pdf.attachover a genuine/api/wsWebSocket (tui_gateway.ws.handle_ws→dispatch— the exact dashboard path), and hitGET /api/mediaover authenticated HTTP. 9/9 live checks passed: attach → gateway disk → queue, gateway-path → data-URL display with byte-identical round-trip, out-of-root 403, symlink-escape 403, non-image 415, missing 404, bad-auth 401.Credit
Consolidates the work of @maxtrigify (#38876), @Justlrnal4 (#40317), @ccook1963 (#21908), and @RyTsYdUp (#39437), preserved via
Co-authored-bytrailers on the commit. Each source PR will be closed pointing here.Infographic