fix(desktop+dashboard): upload composer attachments to the remote backend instead of passing local paths by Justlrnal4 · Pull Request #40317 · NousResearch/hermes-agent

Justlrnal4 · 2026-06-06T04:55:17Z

Problem

Composer images are written to the desktop machine and referenced by local path.
When the desktop drives a remote dashboard, that local path is unreadable on
the backend host, so image.attach fails and the image is dropped (tui_gateway
logs a skipped unreadable path).

Closes #40316

Scope

The new endpoint is general (images plus a small document allow-list, for
future/adjacent use). This PR wires the Desktop recovery path for image
composer attachments only (kind === 'image'); documents are a follow-up.

Changes

New POST /api/attachments (dashboard). Accepts JSON
{ data_url, filename?, mime_type? } — a base64 data URL, not multipart,
mirroring the existing /api/audio/transcribe convention. Protected by the
dashboard auth middleware because the route is not in the public-paths
allow-list. Caches bytes via the existing cache_image_from_bytes /
cache_document_from_bytes, returns a backend-readable path; 25 MB cap +
magic-byte validation; image + small document allow-list (not arbitrary bytes).
Desktop submit (syncImageAttachmentsForSubmit). Attaches by local path
first. The successful local-session fast path is unchanged; the upload
fallback only runs after image.attach cannot read the original path —
then it reads the bytes (readFileDataUrl), uploads via the existing
profile-scoped api proxy, and retries image.attach with the returned
backend path.
No Electron-main, preload, or tui_gateway changes — reuses existing IPC
(readFileDataUrl, api) and the existing image.attach round-trip.

Tests

Backend: new tests/hermes_cli/test_attachment_upload.py (6 cases — caches on
backend host, rejects raw client paths / bad base64 / unsupported MIME /
non-image bytes mislabeled as image / oversize).
Desktop: uploadAttachment unit test (apps/desktop/src/hermes.upload.test.ts).
tsc -b type-check passes; the full vitest suite shows no new failures versus
the base commit.
A full syncImageAttachmentsForSubmit integration test is recommended as a
follow-up (the existing hook test is environment-flaky without an Electron
runtime).

Compatibility

The successful local-session fast path is unchanged; the upload path runs only
when the backend cannot read the original path.

Follow-ups (not in this PR)

Detect remote upfront to skip the one doomed first image.attach.
Wire document attachments through the desktop recovery path.
Warn instead of silently dropping an unreadable attachment in tui_gateway.

Composer images are written to the desktop machine local filesystem and referenced by path. When the desktop drives a REMOTE dashboard, that local path is unreadable by the backend on the other host, so image.attach fails and the image is dropped (tui_gateway logs a skipped unreadable path). Add POST /api/attachments: a JSON {data_url, filename, mime_type} upload (base64 data URL, mirroring /api/audio/transcribe), gated by the dashboard auth middleware, accepting an image plus a small document allow-list, caching bytes via cache_image_from_bytes / cache_document_from_bytes and returning a backend-readable path. On submit, syncImageAttachmentsForSubmit attaches by local path first (local sessions keep the unchanged fast path); on an unreadable-path failure it uploads the bytes and retries with the backend path. No Electron-main, preload, or tui_gateway changes required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…splay gateway images over the network Desktop connected to a remote gateway can now attach images and PDFs and display agent-written images. Previously the desktop passed a LOCAL file path to image.attach; on a remote gateway that path doesn't exist, so the image was silently dropped ("skipped unreadable path") and the vision model never saw it. The reverse direction was also broken — images the agent wrote on the gateway rendered as dead links in the remote client. Gateway (tui_gateway/server.py): - image.attach_bytes: base64 byte upload written into the gateway's own images dir and queued via the existing native-image-attach pipeline. Magic-byte extension sniffing, data-URL prefix + whitespace tolerance, 25 MB cap, structured error codes. Accepts content_base64/filename (canonical) and data/ext (older-desktop aliases). - pdf.attach: renders each page to PNG via pdftoppm (poppler-utils) at 150 DPI and queues the pages as images; 50 MB / 25-page caps. Accepts host path or base64 upload. - Shared helpers (_decode_attach_base64, _sniff_image_ext, _queue_attached_image) so the two methods and the existing image.attach don't duplicate logic. Gateway (hermes_cli/web_server.py): - GET /api/media: returns a gateway-local image as a base64 data URL so remote clients can display it. Auth-gated like every /api route, extension allowlist + size cap, AND confined to the gateway's own media roots (images/screenshots/cache, resolved symlink-safe) so an authed caller can't read image-extension files anywhere on disk. Desktop (apps/desktop): - syncImageAttachmentsForSubmit uploads bytes via image.attach_bytes when the connection mode is 'remote'; the local fast path is unchanged. - media.ts gains isRemoteGateway() + gatewayMediaDataUrl(); directive-text and markdown-text fetch images over /api/media in remote mode. Consolidates the competing remote-media PRs (#38876, #40317, #21908, #39437) into one coherent implementation, taking the strongest parts of each and adding shared-helper cleanup plus the /api/media root-confinement hardening on top. The per-profile gateway switching from #38876 is intentionally left out as a separable feature. TUI file uploads (#40492) remain a separate surface. Tested: 11 new tui_gateway tests + 5 /api/media endpoint tests + desktop media.remote unit tests; full tui_gateway + web_server suites green (472 passed); tsc -b clean; E2E verified the full attach→disk→queue and gateway-path→data-URL display round-trip plus the out-of-root security block. Co-authored-by: Max Mitcham <maxmitcham@mac.home> Co-authored-by: Justlrnal4 <Justlrnal4@users.noreply.github.com> Co-authored-by: Chris Cook <ccook@nvms.com> Co-authored-by: Thomas Paquette <thomas.paquette@gmail.com>

teknium1 · 2026-06-07T17:06:23Z

Merged via #41336 (commit 16786f3 on main).

The remote-media work from this cluster was consolidated into one coherent implementation — image.attach_bytes + pdf.attach over the tui_gateway, root-confined GET /api/media for display, and the desktop remote-mode wiring. Your contribution was cherry-picked with your authorship preserved via a Co-authored-by trailer on the merge commit.

Verified live end-to-end over the real dashboard stack (real /api/ws WebSocket + authenticated HTTP): attach → gateway disk → queue, gateway-path → data-URL display with byte-identical round-trip, plus the security blocks (403 out-of-root, 403 symlink-escape, 415 non-image, 404 missing, 401 bad-auth). Thanks for the work!

…splay gateway images over the network Desktop connected to a remote gateway can now attach images and PDFs and display agent-written images. Previously the desktop passed a LOCAL file path to image.attach; on a remote gateway that path doesn't exist, so the image was silently dropped ("skipped unreadable path") and the vision model never saw it. The reverse direction was also broken — images the agent wrote on the gateway rendered as dead links in the remote client. Gateway (tui_gateway/server.py): - image.attach_bytes: base64 byte upload written into the gateway's own images dir and queued via the existing native-image-attach pipeline. Magic-byte extension sniffing, data-URL prefix + whitespace tolerance, 25 MB cap, structured error codes. Accepts content_base64/filename (canonical) and data/ext (older-desktop aliases). - pdf.attach: renders each page to PNG via pdftoppm (poppler-utils) at 150 DPI and queues the pages as images; 50 MB / 25-page caps. Accepts host path or base64 upload. - Shared helpers (_decode_attach_base64, _sniff_image_ext, _queue_attached_image) so the two methods and the existing image.attach don't duplicate logic. Gateway (hermes_cli/web_server.py): - GET /api/media: returns a gateway-local image as a base64 data URL so remote clients can display it. Auth-gated like every /api route, extension allowlist + size cap, AND confined to the gateway's own media roots (images/screenshots/cache, resolved symlink-safe) so an authed caller can't read image-extension files anywhere on disk. Desktop (apps/desktop): - syncImageAttachmentsForSubmit uploads bytes via image.attach_bytes when the connection mode is 'remote'; the local fast path is unchanged. - media.ts gains isRemoteGateway() + gatewayMediaDataUrl(); directive-text and markdown-text fetch images over /api/media in remote mode. Consolidates the competing remote-media PRs (NousResearch#38876, NousResearch#40317, NousResearch#21908, NousResearch#39437) into one coherent implementation, taking the strongest parts of each and adding shared-helper cleanup plus the /api/media root-confinement hardening on top. The per-profile gateway switching from NousResearch#38876 is intentionally left out as a separable feature. TUI file uploads (NousResearch#40492) remain a separate surface. Tested: 11 new tui_gateway tests + 5 /api/media endpoint tests + desktop media.remote unit tests; full tui_gateway + web_server suites green (472 passed); tsc -b clean; E2E verified the full attach→disk→queue and gateway-path→data-URL display round-trip plus the out-of-root security block. Co-authored-by: Max Mitcham <maxmitcham@mac.home> Co-authored-by: Justlrnal4 <Justlrnal4@users.noreply.github.com> Co-authored-by: Chris Cook <ccook@nvms.com> Co-authored-by: Thomas Paquette <thomas.paquette@gmail.com>

…splay gateway images over the network Desktop connected to a remote gateway can now attach images and PDFs and display agent-written images. Previously the desktop passed a LOCAL file path to image.attach; on a remote gateway that path doesn't exist, so the image was silently dropped ("skipped unreadable path") and the vision model never saw it. The reverse direction was also broken — images the agent wrote on the gateway rendered as dead links in the remote client. Gateway (tui_gateway/server.py): - image.attach_bytes: base64 byte upload written into the gateway's own images dir and queued via the existing native-image-attach pipeline. Magic-byte extension sniffing, data-URL prefix + whitespace tolerance, 25 MB cap, structured error codes. Accepts content_base64/filename (canonical) and data/ext (older-desktop aliases). - pdf.attach: renders each page to PNG via pdftoppm (poppler-utils) at 150 DPI and queues the pages as images; 50 MB / 25-page caps. Accepts host path or base64 upload. - Shared helpers (_decode_attach_base64, _sniff_image_ext, _queue_attached_image) so the two methods and the existing image.attach don't duplicate logic. Gateway (hermes_cli/web_server.py): - GET /api/media: returns a gateway-local image as a base64 data URL so remote clients can display it. Auth-gated like every /api route, extension allowlist + size cap, AND confined to the gateway's own media roots (images/screenshots/cache, resolved symlink-safe) so an authed caller can't read image-extension files anywhere on disk. Desktop (apps/desktop): - syncImageAttachmentsForSubmit uploads bytes via image.attach_bytes when the connection mode is 'remote'; the local fast path is unchanged. - media.ts gains isRemoteGateway() + gatewayMediaDataUrl(); directive-text and markdown-text fetch images over /api/media in remote mode. Consolidates the competing remote-media PRs (#38876, #40317, #21908, #39437) into one coherent implementation, taking the strongest parts of each and adding shared-helper cleanup plus the /api/media root-confinement hardening on top. The per-profile gateway switching from #38876 is intentionally left out as a separable feature. TUI file uploads (#40492) remain a separate surface. Tested: 11 new tui_gateway tests + 5 /api/media endpoint tests + desktop media.remote unit tests; full tui_gateway + web_server suites green (472 passed); tsc -b clean; E2E verified the full attach→disk→queue and gateway-path→data-URL display round-trip plus the out-of-root security block. Co-authored-by: Max Mitcham <maxmitcham@mac.home> Co-authored-by: Justlrnal4 <Justlrnal4@users.noreply.github.com> Co-authored-by: Chris Cook <ccook@nvms.com> Co-authored-by: Thomas Paquette <thomas.paquette@gmail.com>

Justlrnal4 force-pushed the fix/remote-desktop-attachment-upload branch from 9feacef to 8b304ab Compare June 6, 2026 05:25

alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard tool/vision Vision analysis and image generation labels Jun 6, 2026

alt-glitch mentioned this pull request Jun 6, 2026

Remote desktop: composer image attachments are sent as local paths, unreadable by the remote backend #40316

Open

This was referenced Jun 7, 2026

feat(desktop+gateway): remote media relay — attach images/PDFs & display gateway images over the network #41203

Closed

feat(desktop+gateway): remote media relay — attach images/PDFs & display gateway images over the network #41336

Merged

teknium1 closed this Jun 7, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(desktop+dashboard): upload composer attachments to the remote backend instead of passing local paths#40317

fix(desktop+dashboard): upload composer attachments to the remote backend instead of passing local paths#40317
Justlrnal4 wants to merge 1 commit into
NousResearch:mainfrom
Justlrnal4:fix/remote-desktop-attachment-upload

Justlrnal4 commented Jun 6, 2026

Uh oh!

teknium1 commented Jun 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Justlrnal4 commented Jun 6, 2026

Problem

Scope

Changes

Tests

Compatibility

Follow-ups (not in this PR)

Uh oh!

teknium1 commented Jun 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants