Skip to content

feat(desktop): remote-mode image attach/display + per-profile gateway switching#38876

Closed
maxtrigify wants to merge 1 commit into
NousResearch:mainfrom
maxtrigify:desktop-remote-fixes
Closed

feat(desktop): remote-mode image attach/display + per-profile gateway switching#38876
maxtrigify wants to merge 1 commit into
NousResearch:mainfrom
maxtrigify:desktop-remote-fixes

Conversation

@maxtrigify

@maxtrigify maxtrigify commented Jun 4, 2026

Copy link
Copy Markdown

Three remote-mode improvements for the desktop app, each shipped with the gateway endpoint it needs so the feature works on a fresh deploy (no out-of-tree dependencies).

1. Remote image attach

When the desktop is connected to a remote gateway, attaching an image used to send a local file path — which doesn't exist on the gateway machine (the gateway then split the path on the first space, e.g. …/Library/Application). Now, in remote mode, the client uploads the image bytes via a new image.attach_bytes RPC (tui_gateway/server.py); local same-machine mode keeps the path-based image.attach untouched. The new RPC decodes base64, strips any data-URL prefix, sanitizes the extension, caps at 25 MB, writes into the gateway's own images dir, and returns the same response shape as image.attach.

2. Remote image display

Images the agent writes to the gateway's disk can't be read by a remote client, so they rendered as dead "Open …png" links. New GET /api/media (hermes_cli/web_server.py) returns an image file as a base64 data URL — auth-gated by the session token like every /api route, restricted to an image-extension allowlist, size-capped. The desktop fetches via it in remote mode (media.ts, markdown-text.tsx, directive-text.tsx), with graceful fallback to the existing chip when unavailable. Also fixes inline images in the web dashboard when viewed remotely.

3. Per-profile gateway switching

connection.json now holds a remotes[] list (+ activeRemoteId), auto-migrated from the single remote, each entry carrying its auth mode (token or oauth). Settings → Gateway manages the list (add / remove / connect), and the titlebar Profiles menu switches between them with one click (reconnects). Profile name and auth mode are auto-detected from the gateway's /api/status (hermes_home → profile; auth_required → oauth). Integrates with the existing OAuth connection layer — token gateways and OAuth-gated gateways are both supported per connection.

Notes

  • All three are independent; happy to split into separate PRs if preferred.
  • Verified: tsc -b clean, eslint clean (no new findings), py_compile clean on both gateway files, full desktop build succeeds, and the running build exercises all three.

🤖 Generated with Claude Code

@claude
feat(desktop): remote-mode image attach/display + per-profile gateway…
7759141 
… switching

Three remote-mode improvements for the desktop app, plus the gateway endpoints
they need (so the feature is self-contained on a fresh deploy):

1. Remote image attach — when connected to a remote gateway, upload image BYTES
   via a new `image.attach_bytes` RPC (tui_gateway/server.py) instead of sending
   a local path the gateway can't read. Local mode keeps path-based image.attach.

2. Remote image display — agent-created images live on the gateway's disk, which
   a remote client can't read. New `GET /api/media` (web_server.py) serves an
   image file as a base64 data URL (auth-gated, extension-allowlisted, size-
   capped); the desktop fetches via it in remote mode, with graceful fallback.

3. Per-profile gateway switching — connection.json now holds a `remotes[]` list
   (+ activeRemoteId, auto-migrated from the single remote), each carrying its
   auth mode (token or OAuth). Settings → Gateway manages them; the titlebar
   Profiles menu switches between them. Profile + auth mode are auto-detected
   from the gateway's /api/status. Integrates with the OAuth connection layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/feature New feature or request comp/gateway Gateway runner, session dispatch, delivery comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have labels Jun 4, 2026
This was referenced Jun 7, 2026
Closed
Merged
teknium1 added a commit that referenced this pull request Jun 7, 2026
@teknium1 @Justlrnal4
@ccook1963 @RyTsYdUp
feat(desktop+gateway): remote media relay — attach images/PDFs and di…
16786f3 
…splay gateway images over the network

Desktop connected to a remote gateway can now attach images and PDFs and
display agent-written images. Previously the desktop passed a LOCAL file path
to image.attach; on a remote gateway that path doesn't exist, so the image was
silently dropped ("skipped unreadable path") and the vision model never saw it.
The reverse direction was also broken — images the agent wrote on the gateway
rendered as dead links in the remote client.

Gateway (tui_gateway/server.py):
- image.attach_bytes: base64 byte upload written into the gateway's own images
  dir and queued via the existing native-image-attach pipeline. Magic-byte
  extension sniffing, data-URL prefix + whitespace tolerance, 25 MB cap,
  structured error codes. Accepts content_base64/filename (canonical) and
  data/ext (older-desktop aliases).
- pdf.attach: renders each page to PNG via pdftoppm (poppler-utils) at 150 DPI
  and queues the pages as images; 50 MB / 25-page caps. Accepts host path or
  base64 upload.
- Shared helpers (_decode_attach_base64, _sniff_image_ext, _queue_attached_image)
  so the two methods and the existing image.attach don't duplicate logic.

Gateway (hermes_cli/web_server.py):
- GET /api/media: returns a gateway-local image as a base64 data URL so remote
  clients can display it. Auth-gated like every /api route, extension
  allowlist + size cap, AND confined to the gateway's own media roots
  (images/screenshots/cache, resolved symlink-safe) so an authed caller can't
  read image-extension files anywhere on disk.

Desktop (apps/desktop):
- syncImageAttachmentsForSubmit uploads bytes via image.attach_bytes when the
  connection mode is 'remote'; the local fast path is unchanged.
- media.ts gains isRemoteGateway() + gatewayMediaDataUrl(); directive-text and
  markdown-text fetch images over /api/media in remote mode.

Consolidates the competing remote-media PRs (#38876, #40317, #21908, #39437)
into one coherent implementation, taking the strongest parts of each and adding
shared-helper cleanup plus the /api/media root-confinement hardening on top.
The per-profile gateway switching from #38876 is intentionally left out as a
separable feature. TUI file uploads (#40492) remain a separate surface.

Tested: 11 new tui_gateway tests + 5 /api/media endpoint tests + desktop
media.remote unit tests; full tui_gateway + web_server suites green (472
passed); tsc -b clean; E2E verified the full attach→disk→queue and
gateway-path→data-URL display round-trip plus the out-of-root security block.

Co-authored-by: Max Mitcham <maxmitcham@mac.home>
Co-authored-by: Justlrnal4 <Justlrnal4@users.noreply.github.com>
Co-authored-by: Chris Cook <ccook@nvms.com>
Co-authored-by: Thomas Paquette <thomas.paquette@gmail.com>
@teknium1

teknium1 commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Merged via #41336 (commit 16786f3 on main).

The remote-media work from this cluster was consolidated into one coherent implementation — image.attach_bytes + pdf.attach over the tui_gateway, root-confined GET /api/media for display, and the desktop remote-mode wiring. Your contribution was cherry-picked with your authorship preserved via a Co-authored-by trailer on the merge commit.

Verified live end-to-end over the real dashboard stack (real /api/ws WebSocket + authenticated HTTP): attach → gateway disk → queue, gateway-path → data-URL display with byte-identical round-trip, plus the security blocks (403 out-of-root, 403 symlink-escape, 415 non-image, 404 missing, 401 bad-auth). Thanks for the work!

@teknium1 teknium1 closed this Jun 7, 2026
changman pushed a commit to changman/hermes-agent that referenced this pull request Jun 10, 2026
@teknium1 @Justlrnal4
@ccook1963 @RyTsYdUp @changman
feat(desktop+gateway): remote media relay — attach images/PDFs and di…
4fb45d7 
…splay gateway images over the network

Desktop connected to a remote gateway can now attach images and PDFs and
display agent-written images. Previously the desktop passed a LOCAL file path
to image.attach; on a remote gateway that path doesn't exist, so the image was
silently dropped ("skipped unreadable path") and the vision model never saw it.
The reverse direction was also broken — images the agent wrote on the gateway
rendered as dead links in the remote client.

Gateway (tui_gateway/server.py):
- image.attach_bytes: base64 byte upload written into the gateway's own images
  dir and queued via the existing native-image-attach pipeline. Magic-byte
  extension sniffing, data-URL prefix + whitespace tolerance, 25 MB cap,
  structured error codes. Accepts content_base64/filename (canonical) and
  data/ext (older-desktop aliases).
- pdf.attach: renders each page to PNG via pdftoppm (poppler-utils) at 150 DPI
  and queues the pages as images; 50 MB / 25-page caps. Accepts host path or
  base64 upload.
- Shared helpers (_decode_attach_base64, _sniff_image_ext, _queue_attached_image)
  so the two methods and the existing image.attach don't duplicate logic.

Gateway (hermes_cli/web_server.py):
- GET /api/media: returns a gateway-local image as a base64 data URL so remote
  clients can display it. Auth-gated like every /api route, extension
  allowlist + size cap, AND confined to the gateway's own media roots
  (images/screenshots/cache, resolved symlink-safe) so an authed caller can't
  read image-extension files anywhere on disk.

Desktop (apps/desktop):
- syncImageAttachmentsForSubmit uploads bytes via image.attach_bytes when the
  connection mode is 'remote'; the local fast path is unchanged.
- media.ts gains isRemoteGateway() + gatewayMediaDataUrl(); directive-text and
  markdown-text fetch images over /api/media in remote mode.

Consolidates the competing remote-media PRs (NousResearch#38876, NousResearch#40317, NousResearch#21908, NousResearch#39437)
into one coherent implementation, taking the strongest parts of each and adding
shared-helper cleanup plus the /api/media root-confinement hardening on top.
The per-profile gateway switching from NousResearch#38876 is intentionally left out as a
separable feature. TUI file uploads (NousResearch#40492) remain a separate surface.

Tested: 11 new tui_gateway tests + 5 /api/media endpoint tests + desktop
media.remote unit tests; full tui_gateway + web_server suites green (472
passed); tsc -b clean; E2E verified the full attach→disk→queue and
gateway-path→data-URL display round-trip plus the out-of-root security block.

Co-authored-by: Max Mitcham <maxmitcham@mac.home>
Co-authored-by: Justlrnal4 <Justlrnal4@users.noreply.github.com>
Co-authored-by: Chris Cook <ccook@nvms.com>
Co-authored-by: Thomas Paquette <thomas.paquette@gmail.com>
alt-glitch pushed a commit that referenced this pull request Jun 14, 2026
@teknium1 @Justlrnal4
@ccook1963 @RyTsYdUp
feat(desktop+gateway): remote media relay — attach images/PDFs and di…
f95626b 
…splay gateway images over the network

Desktop connected to a remote gateway can now attach images and PDFs and
display agent-written images. Previously the desktop passed a LOCAL file path
to image.attach; on a remote gateway that path doesn't exist, so the image was
silently dropped ("skipped unreadable path") and the vision model never saw it.
The reverse direction was also broken — images the agent wrote on the gateway
rendered as dead links in the remote client.

Gateway (tui_gateway/server.py):
- image.attach_bytes: base64 byte upload written into the gateway's own images
  dir and queued via the existing native-image-attach pipeline. Magic-byte
  extension sniffing, data-URL prefix + whitespace tolerance, 25 MB cap,
  structured error codes. Accepts content_base64/filename (canonical) and
  data/ext (older-desktop aliases).
- pdf.attach: renders each page to PNG via pdftoppm (poppler-utils) at 150 DPI
  and queues the pages as images; 50 MB / 25-page caps. Accepts host path or
  base64 upload.
- Shared helpers (_decode_attach_base64, _sniff_image_ext, _queue_attached_image)
  so the two methods and the existing image.attach don't duplicate logic.

Gateway (hermes_cli/web_server.py):
- GET /api/media: returns a gateway-local image as a base64 data URL so remote
  clients can display it. Auth-gated like every /api route, extension
  allowlist + size cap, AND confined to the gateway's own media roots
  (images/screenshots/cache, resolved symlink-safe) so an authed caller can't
  read image-extension files anywhere on disk.

Desktop (apps/desktop):
- syncImageAttachmentsForSubmit uploads bytes via image.attach_bytes when the
  connection mode is 'remote'; the local fast path is unchanged.
- media.ts gains isRemoteGateway() + gatewayMediaDataUrl(); directive-text and
  markdown-text fetch images over /api/media in remote mode.

Consolidates the competing remote-media PRs (#38876, #40317, #21908, #39437)
into one coherent implementation, taking the strongest parts of each and adding
shared-helper cleanup plus the /api/media root-confinement hardening on top.
The per-profile gateway switching from #38876 is intentionally left out as a
separable feature. TUI file uploads (#40492) remain a separate surface.

Tested: 11 new tui_gateway tests + 5 /api/media endpoint tests + desktop
media.remote unit tests; full tui_gateway + web_server suites green (472
passed); tsc -b clean; E2E verified the full attach→disk→queue and
gateway-path→data-URL display round-trip plus the out-of-root security block.

Co-authored-by: Max Mitcham <maxmitcham@mac.home>
Co-authored-by: Justlrnal4 <Justlrnal4@users.noreply.github.com>
Co-authored-by: Chris Cook <ccook@nvms.com>
Co-authored-by: Thomas Paquette <thomas.paquette@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have type/feature New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@maxtrigify @teknium1 @alt-glitch