β¨ feat(hetero-agent): support multimodal input (text + images) across CLI / shared spawn / desktop#14433
Merged
Merged
Conversation
β¦awn / desktop `spawnAgent` and `lh hetero exec` could only take a flat string prompt, so attaching images required bypassing the shared layer (which is what desktop actually did). This adds a unified `AgentPromptInput` shape β string sugar or an array of text/image content blocks β and lifts image handling into the shared `@lobechat/heterogeneous-agents/spawn/input` module. Image sources accept URL (with optional id for cache dedupe), local path, or inline base64. The shared `normalizeImage` fetches/reads/decodes, with optional on-disk caching keyed by `sha256(id || url)`. `materializeImageToPath` writes buffers to a cache dir (used by Codex `--image <path>`), with byte- signature sniffing fallback when MIME is generic. `buildAgentInput` is the single source of truth for per-agent serialization: Claude Code receives base64 image blocks inline in stream-json; Codex receives text on stdin + repeatable `--image <path>` flags. CLI gets three input modes: `--prompt <text>` + `--image <path|url|data:>` (repeatable), `--input-json <file|->` for full content-block JSON, and stdin auto-detection (JSON vs plain text by first non-whitespace character). Mutually-exclusive flag combinations error early. Desktop's `HeterogeneousAgentCtr` drops ~100 lines of duplicated cache / sniffing code; helpers (`buildStreamJsonInput`, `resolveCliImagePaths`) become thin wrappers around the shared functions. Driver interface and IPC contract are unchanged. `spawnAgent` is now async (image normalization fetches/reads before spawn). Verified end-to-end: `lh hetero exec --type claude-code --prompt ... --image red.png` β CC replied "I see a solid red color." `--input-json` mode also verified. 28/28 desktop tests, 11/11 CLI hetero tests, 22/22 spawn package tests pass. Refs LOBE-8523 (phase 1a follow-up before phase 1b ingest). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop workspace deps on @lobechat/web-crawler and @lobechat/python-interpreter from @lobechat/types by inlining CrawlSuccessResult / CrawlErrorResult / CrawlUniformResult and PythonOutput / PythonResult into the relevant tool type modules. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
π‘ Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ceaf114182
βΉοΈ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with π.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codecov Reportβ
All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## canary #14433 +/- ##
=========================================
Coverage 68.69% 68.69%
=========================================
Files 2499 2499
Lines 214321 214321
Branches 22469 26091 +3622
=========================================
Hits 147219 147219
Misses 66959 66959
Partials 143 143
Flags with carried forward coverage won't be shown. Click here to find out more.
π New features to boost your workflow:
|
β¦nder header Show the human-readable `description` arg in the gh tool's collapsed inspector chip and result-card header when provided; fall back to the extracted subcommand. Full command is still visible in the expanded Command code block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
β¦sync spawnAgent failures Two issues raised on PR #14433 review: **P1 β generic Content-Type bypassed sniffing in normalizeImage** `fetchUrlImage` accepted any non-empty `Content-Type` as the final `mediaType`, so CDN responses defaulting to `application/octet-stream` (or `text/plain`) skipped URL/byte-based detection and forwarded an unrecognized type into Claude Code's stream-json `media_type` field β Anthropic rejects those even when the bytes are a valid PNG/JPEG. The same flaw existed for base64 sources whose declared `mediaType` was generic. Introduce `pickImageMediaType(headerType, url, buffer)`: the header value is preferred only when it's a recognized `image/*` type we know how to extension- map; otherwise it falls through to URL extension hint β byte-signature sniff β raw header β `image/png` final fallback. Applied uniformly to URL fetch, URL cache hit, and base64 decode paths. Path sources are unchanged (their "header" is the file extension, which is already authoritative when present). **P2 β async spawnAgent rejections crashed the CLI** `spawnAgent` is now async and can reject during image normalization (missing local `--image` path, fetch failure, decode error). The CLI awaited it outside any try/catch, so user-input errors surfaced as unhandled rejections with stack traces instead of the friendly `log.error + process.exit` path used for prompt validation. Wrap the `await spawnAgent(...)` in try/catch, log the error message, exit 1 (matching the existing "Stream error from agent process" convention). **Tests** - `buildAgentInput.test.ts`: 3 new tests covering octet-stream URL Content-Type β byte sniff, octet-stream base64 declared type β byte sniff, generic header + URL extension hint preferred over header. - `hetero.test.ts`: 1 new test verifying spawnAgent rejection produces clean `exit(1)` instead of an unhandled rejection. Manually verified: `lh hetero exec --image /tmp/does-not-exist.png` β `[ERROR] Failed to start agent: ENOENT: no such file or directoryβ¦` + exit 1 Refs LOBE-8523. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Innei
added a commit
that referenced
this pull request
May 9, 2026
# π LobeHub Release (20260509) **Release Date:** May 9, 2026 **Since v2.1.56:** 236 merged PRs Β· 19 contributors > Agent Task System reaches general availability, the Agent Signal pipeline runs nightly self-review with skill-aware policies, the heterogeneous-agent runtime crosses replica boundaries, inline documents become a first-class context source, and bot platforms expand across Messager, Line, and Telegram. --- ## β¨ Highlights - **Agent Task System (GA)** β End-to-end task execution platform: templates, tracking, comment tools, parent reassignment, scheduled cron, and dependency-ordered batch runs. (#14540, #14515, #14517, #14272, #14246, #14418, #14403, #14488) - **Agent Signal nightly self-review** β Wired self-review loop with prompt + DB support, exponential-backoff retry on receipt listing, skill-aware policy, and improved skill-intent detection. (#14543, #14542, #14281, #14409, #14526, #14437) - **Inline documents in KB tool** β BM25 search and `docs_*` read for inline document grounding; agent documents usable as VFS. (#14494, #14222) - **Inline agent cards in chat** β `lobeAgents` markdown tag renders agent profile cards inline; clickable card after `createAgent`. (#14495, #14493) - **Heterogeneous agent runtime** β Cloud hetero exec pipeline steps 3+4 land, persistence recovers across Vercel replicas, server-side ingest/finish handler, and `lh hetero exec` CLI. (#14486, #14539, #14444, #14431) - **Bot platforms expand** β Messager, Line, DM pair policy, and messenger DB tables; Telegram API path restored. (#14442, #14207, #14211, #14496, #14519) - **Visual analysis tool** β New visual understanding tool, with trigger tracking and flattened schema. (#14378, #14399, #14550) - **DeepSeek V4 Pro as OSS default** β OSS deployments ship with DeepSeek V4 Pro by default; DeepSeek Anthropic runtime supported. (#14555, #14312) --- ## ποΈ Core Agent & Architecture ### Agent Task System - **Task System GA** β End-to-end execution platform now available. (#14540) - **Templates, comments, reparenting** β Template tracking, comment tools, and parent reassignment. (#14515, #14517, #14488) - **Cron + dependency-ordered runs** β Scheduled status with cron editor and dependency-ordered subtask batches. (#14246, #14418, #14272) - **Inspector + chip UI + batch tasks** β Task Inspector/Render registry, batch `createTasks`/`runTasks`, and chip-based agent-documents inspector. (#14403, #14404) - **Recommend templates regardless of brief count** β Recommendations no longer suppressed when briefs are sparse. (#14508) - **Scheduling resilience** β Manual run no longer eats next scheduled tick; recurring tasks survive brief resolution. (#14304, #14348) - **Brief synthesis** β Auto-synthesize topic briefs; brief actions revamp; mute resolved-brief icon on home. (#14324, #14228, #14452) - **Task list & detail polish** β Topic operation ID exposed; task drawer Gateway reconnect. (#14282) ### Agent Signal pipeline - **Nightly self-review wired** β Prompt + DB support for the self-review loop. (#14543) - **Self-review activities push to briefs** β Activities during nightly self-reflection now create briefs. (#14437) - **Skill management policy** β New policy for Skill management running inside Agent Signal. (#14281) - **Skill intent detection & routing** β Improved detection plus direct intent handling when `hintIsSkill`. (#14409, #14526) - **Document tool outcome rendering** β Decision view restores missing document tool outcomes. (#14534) - **Exponential backoff retry** β Listing signal receipts retries with jittered backoff. (#14542) - **Easier-to-use signals** β Structural simplification + recent-activities surface for receipts. (#14290, #14326, #14407) ### Heterogeneous agent runtime - **Cloud hetero exec pipeline (steps 3 + 4)** β Refactor lands the next two stages of the cloud hetero agent execution pipeline. (#14486) - **Persistence recovery on Vercel** β Hetero state recovered across replica boundaries. (#14539) - **Server-side ingest/finish + persistence** β `aiAgent.heteroIngest` / `heteroFinish` handlers. (#14444) - **`lh hetero exec` CLI** β Standalone heterogeneous agent runs from CLI. (#14431) - **Gateway round-trip loading** β `execAgentTask` keeps the input box in loading state through the full round-trip. (#14503) - **Provider SDK type routing** β Provider routing now respects SDK type. (#14520) - **DeepSeek reasoning preserved** β `reasoning_content` preserved in OpenAI-compatible runtime for DeepSeek models. (#14546) ### Knowledge & inline docs - **KB tool BM25 + docs read** β BM25 search and `docs_*` read integrated for inline documents. (#14494) - **Agent documents as VFS** β FS-compatible output for agent documents. (#14222) - **`lobeAgents` markdown tag** β Inline agent cards rendered from a markdown tag. (#14495) - **Clickable agent card after `createAgent`** β Mentions and recommendations become clickable. (#14493) - **ExplorerTree** β Generic tree component built on `@pierre/trees` for reusable explorer surfaces. (#14094) - **Local file mention snapshots** β Mentions can now snapshot local files. (#14278) ### Architecture - **Agent Hono routes** β New agent routes added on Hono. (#14535) - **`/api/agent` migrated to Hono** β Remaining `/api/agent` routes finish their migration. (#14478) - **Agent marketplace merged into web-onboarding** β Reduces package fragmentation. (#14514) - **Producer pipeline extracted** β Shared package for the producer pipeline. (#14425) - **`agentDispatcher.selectRuntimeType`** β New runtime selection abstraction. (#14428) - **pnpm v11 migration** β Workspace consolidated. (#14316) - **Browser-compatible frontmatter parser** β Replaces `gray-matter`. (#14435) --- ## π± Platforms & Integrations - **Messager support** β New messager package wired into the chat surface. (#14442) - **Messenger DB tables** β IM bot integration gains its persistence layer. (#14496) - **Line bot** β Initial Line support and downstream optimization. (#14207, #14448) - **DM pair policy** β Group/DM pair-based delivery. (#14211) - **Telegram API restored** β Missing Telegram API path reconnected. (#14519) - **xAI Responses tools stabilized** β Plus unsupported parameter handling. (#14462, #14445) - **Volcengine websearch via ResponseAPI** β Built-in websearch for Volcengine. (#14216) --- ## π€ Models & Providers - **DeepSeek V4 Pro default for OSS** β OSS distribution defaults to DeepSeek V4 Pro. (#14555) - **DeepSeek Anthropic runtime** β Anthropic-shape runtime support for DeepSeek. (#14312) - **GPT-5.5 / GPT-5.5 Pro** β New OpenAI tier. (#14142) - **Grok 4.20 / Grok 4.3 / LobeHub-hosted Grok 4.3** β (#14253, #14382, #14446) - **Gemma 4 + provider settings normalization** β (#13313) - **gpt-image-2 + step-image-edit-2** β (#14253, #14329) - **Model bank refresh + original-pricing display** β Batch model updates and pricing surfaces. (#14070, #14391) - **Hunyuan migrated to TokenHub for Hy3 Preview** β (#14108) - **Reject lobehub model ids no longer in the bank** β (#14261) - **Hide runtime-only aliases** β Runtime-only model aliases no longer leak into the model picker. (#14552) --- ## π₯οΈ User Experience ### Onboarding - **Shared prefix steps** β Language and privacy extracted as shared prefix steps. (#14538) - **Identity intervention card simplified** β Plus tool result renders cleanup. (#14505, #14506) - **Welcome polish + web-onboarding tool UI** β (#14475) - **Templates fetched from market API** β (#14286) - **Virtual model id for default onboarding model** β (#14311) - **Skip / mode-switch footer behind feature flag** β Footer guarded for desktop and web initialization. (#14560) ### Home & navigation - **Home recents performance** β Recents refresh periodically and inline task status; brief and task-template fetch overhead trimmed. (#14518, #14516) - **Home refactor + skill-connect recommendations** β Restructured home with skill-connect recommendation system. (#14266, #14214) - **Tasks in agent sidebar** β Tasks moved from welcome card into the sidebar list. (#14500) - **Sidebar collapse persists** β Home sidebar collapse state stored. (#14473) - **Agent-specific topic grouping** β Plus improved empty state and agent identity in topic search. (#14225) - **MentionMenu scroll fix** β Mention menu no longer clips inside chat input. (#14533) ### Conversation & chat - **Follow-up chips fill input** β Clicking a follow-up chip now fills the input instead of sending immediately. (#14536) - **Quick-reply chips below assistant messages** β (#14350) - **Inline single-tool assistant group + leading sentence promotion** β (#14244) - **Assistant-group rendering** β Per-segment content overrides flow into MessageContent. (#14504) - **Tool call timer fix** β Timer no longer resets when tool calls collapse or expand. (#14513) - **Streaming re-render reduction** β Reference stabilization and self-subscribing components. (#14470) - **Topic chat drawer feedback input** β (#14392) ### Skills, agents, devtools - **Managed skill folders** β Agent view displays managed skill folders and aligns delete confirmations. (#14553) - **Review tab + bulk git diffs** β New Review tab with bulk diffs; gating uses effective working directory. (#14334, #14512) - **Devtools gallery rebuild** β Plus Review polish, queue-tray images. (#14423) - **Agent mock devtools** β Playback & fixture viewer. (#14436) ### Desktop & CLI - **App tray visibility setting** β (#14463) - **Notification settings in desktop** β (#14491) - **Multimodal input across CLI / shared spawn / desktop** β (#14433) - **CLI bot + userId guide** β (#14258) --- ## π§ Tooling - **Visual analysis tool** β New visual understanding tool with flattened schema. (#14378, #14550) - **GitHub marketplace tool UI** β (#14420) - **Drop "Local" prefix and `____builtin` suffix from tool names** β (#14364, #14289) - **Sanitize provider tool names** β Avoids invalid characters from external providers. (#14510) - **Generation moderation context** β Moderation context passed through the generation pipeline. (#14541) - **Visual analysis trigger tracking** β (#14399) - **Claude thinking signature sanitization** β History signatures sanitized when replaying Claude conversations. (#14499) - **Responses input media sanitization** β Assistant media sanitized in Responses input. (#14497) --- ## π Security & Reliability - **Security:** Removed the `/webapi/proxy` route and dead URL-manifest plugin code to shrink the SSRF surface. (#14549) - **Security:** Sessions revoked after password reset. (#14424) - **Reliability:** Added `prompt_cache_key` to OpenAI chat requests for stable cache hits. (#14349) - **Reliability:** `onFinish` now fires even when the browser tab is backgrounded mid-SSE stream. (#14461) - **Reliability:** Better-auth session refetch preserves user fields rather than overwriting them. (#14531) - **Reliability:** User-memory queries sanitize backticks; user-memory errors now explicitly injected so failures stay visible. (#14524, #14525) - **Reliability:** Auth captcha retries handled; input loading unsticks on `auth_failed` and recoverable `auth_expired`. (#14346, #14419) - **Reliability:** Trace snapshot finalized on error path. (#14440) - **Reliability:** Drop `switchTopic` race under rapid sidebar clicks. (#14115) - **Reliability:** PDF chunking logic fixed to prevent vectorization failure. (#14327) - **Performance:** Marketplace fork uses a batched API for parallel installs. (#14537) - **Performance:** Review tab open latency cut ~9Γ on large dirty trees. (#14338) --- ## π₯ Contributors Huge thanks to **18 contributors** who shipped **236 merged PRs** this cycle. @hezhijie0327 Β· @sxjeru Β· @yueyinqiu Β· @octo-patch Β· @hardy-one Β· @Coooolfan Β· @CanYuanA Β· @BillionClaw Β· @arvinxx Β· @tjx666 Β· @Innei Β· @neko Β· @AmAzing129 Β· @rdmclin2 Β· @lijian Β· @sudongyuer Β· @rivertwilight Β· @cy948 Plus @lobehubbot for i18n and translation maintenance. --- **Full Changelog**: v2.1.56...release/weekly-20260509
github-actions Bot
added a commit
to privadomy/lobe-chat
that referenced
this pull request
May 18, 2026
β¦awn / desktop (lobehub#14433) * β¨ feat(hetero-agent): support multimodal input across CLI / shared spawn / desktop `spawnAgent` and `lh hetero exec` could only take a flat string prompt, so attaching images required bypassing the shared layer (which is what desktop actually did). This adds a unified `AgentPromptInput` shape β string sugar or an array of text/image content blocks β and lifts image handling into the shared `@lobechat/heterogeneous-agents/spawn/input` module. Image sources accept URL (with optional id for cache dedupe), local path, or inline base64. The shared `normalizeImage` fetches/reads/decodes, with optional on-disk caching keyed by `sha256(id || url)`. `materializeImageToPath` writes buffers to a cache dir (used by Codex `--image <path>`), with byte- signature sniffing fallback when MIME is generic. `buildAgentInput` is the single source of truth for per-agent serialization: Claude Code receives base64 image blocks inline in stream-json; Codex receives text on stdin + repeatable `--image <path>` flags. CLI gets three input modes: `--prompt <text>` + `--image <path|url|data:>` (repeatable), `--input-json <file|->` for full content-block JSON, and stdin auto-detection (JSON vs plain text by first non-whitespace character). Mutually-exclusive flag combinations error early. Desktop's `HeterogeneousAgentCtr` drops ~100 lines of duplicated cache / sniffing code; helpers (`buildStreamJsonInput`, `resolveCliImagePaths`) become thin wrappers around the shared functions. Driver interface and IPC contract are unchanged. `spawnAgent` is now async (image normalization fetches/reads before spawn). Verified end-to-end: `lh hetero exec --type claude-code --prompt ... --image red.png` β CC replied "I see a solid red color." `--input-json` mode also verified. 28/28 desktop tests, 11/11 CLI hetero tests, 22/22 spawn package tests pass. Refs LOBE-8523 (phase 1a follow-up before phase 1b ingest). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * π§ chore(cli): include types/model-bank/business-const in workspace Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * β»οΈ refactor(types): inline crawler and python-interpreter types Drop workspace deps on @lobechat/web-crawler and @lobechat/python-interpreter from @lobechat/types by inlining CrawlSuccessResult / CrawlErrorResult / CrawlUniformResult and PythonOutput / PythonResult into the relevant tool type modules. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * π chore(cli): bump @lobehub/cli to 0.0.10 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * π style(github-tool): prefer description over command in inspector/render header Show the human-readable `description` arg in the gh tool's collapsed inspector chip and result-card header when provided; fall back to the extracted subcommand. Full command is still visible in the expanded Command code block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * π fix(hetero-agent): treat generic Content-Type as unknown + handle async spawnAgent failures Two issues raised on PR lobehub#14433 review: **P1 β generic Content-Type bypassed sniffing in normalizeImage** `fetchUrlImage` accepted any non-empty `Content-Type` as the final `mediaType`, so CDN responses defaulting to `application/octet-stream` (or `text/plain`) skipped URL/byte-based detection and forwarded an unrecognized type into Claude Code's stream-json `media_type` field β Anthropic rejects those even when the bytes are a valid PNG/JPEG. The same flaw existed for base64 sources whose declared `mediaType` was generic. Introduce `pickImageMediaType(headerType, url, buffer)`: the header value is preferred only when it's a recognized `image/*` type we know how to extension- map; otherwise it falls through to URL extension hint β byte-signature sniff β raw header β `image/png` final fallback. Applied uniformly to URL fetch, URL cache hit, and base64 decode paths. Path sources are unchanged (their "header" is the file extension, which is already authoritative when present). **P2 β async spawnAgent rejections crashed the CLI** `spawnAgent` is now async and can reject during image normalization (missing local `--image` path, fetch failure, decode error). The CLI awaited it outside any try/catch, so user-input errors surfaced as unhandled rejections with stack traces instead of the friendly `log.error + process.exit` path used for prompt validation. Wrap the `await spawnAgent(...)` in try/catch, log the error message, exit 1 (matching the existing "Stream error from agent process" convention). **Tests** - `buildAgentInput.test.ts`: 3 new tests covering octet-stream URL Content-Type β byte sniff, octet-stream base64 declared type β byte sniff, generic header + URL extension hint preferred over header. - `hetero.test.ts`: 1 new test verifying spawnAgent rejection produces clean `exit(1)` instead of an unhandled rejection. Manually verified: `lh hetero exec --image /tmp/does-not-exist.png` β `[ERROR] Failed to start agent: ENOENT: no such file or directoryβ¦` + exit 1 Refs LOBE-8523. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
github-actions Bot
added a commit
to privadomy/lobe-chat
that referenced
this pull request
May 18, 2026
β¦awn / desktop (lobehub#14433) * β¨ feat(hetero-agent): support multimodal input across CLI / shared spawn / desktop `spawnAgent` and `lh hetero exec` could only take a flat string prompt, so attaching images required bypassing the shared layer (which is what desktop actually did). This adds a unified `AgentPromptInput` shape β string sugar or an array of text/image content blocks β and lifts image handling into the shared `@lobechat/heterogeneous-agents/spawn/input` module. Image sources accept URL (with optional id for cache dedupe), local path, or inline base64. The shared `normalizeImage` fetches/reads/decodes, with optional on-disk caching keyed by `sha256(id || url)`. `materializeImageToPath` writes buffers to a cache dir (used by Codex `--image <path>`), with byte- signature sniffing fallback when MIME is generic. `buildAgentInput` is the single source of truth for per-agent serialization: Claude Code receives base64 image blocks inline in stream-json; Codex receives text on stdin + repeatable `--image <path>` flags. CLI gets three input modes: `--prompt <text>` + `--image <path|url|data:>` (repeatable), `--input-json <file|->` for full content-block JSON, and stdin auto-detection (JSON vs plain text by first non-whitespace character). Mutually-exclusive flag combinations error early. Desktop's `HeterogeneousAgentCtr` drops ~100 lines of duplicated cache / sniffing code; helpers (`buildStreamJsonInput`, `resolveCliImagePaths`) become thin wrappers around the shared functions. Driver interface and IPC contract are unchanged. `spawnAgent` is now async (image normalization fetches/reads before spawn). Verified end-to-end: `lh hetero exec --type claude-code --prompt ... --image red.png` β CC replied "I see a solid red color." `--input-json` mode also verified. 28/28 desktop tests, 11/11 CLI hetero tests, 22/22 spawn package tests pass. Refs LOBE-8523 (phase 1a follow-up before phase 1b ingest). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * π§ chore(cli): include types/model-bank/business-const in workspace Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * β»οΈ refactor(types): inline crawler and python-interpreter types Drop workspace deps on @lobechat/web-crawler and @lobechat/python-interpreter from @lobechat/types by inlining CrawlSuccessResult / CrawlErrorResult / CrawlUniformResult and PythonOutput / PythonResult into the relevant tool type modules. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * π chore(cli): bump @lobehub/cli to 0.0.10 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * π style(github-tool): prefer description over command in inspector/render header Show the human-readable `description` arg in the gh tool's collapsed inspector chip and result-card header when provided; fall back to the extracted subcommand. Full command is still visible in the expanded Command code block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * π fix(hetero-agent): treat generic Content-Type as unknown + handle async spawnAgent failures Two issues raised on PR lobehub#14433 review: **P1 β generic Content-Type bypassed sniffing in normalizeImage** `fetchUrlImage` accepted any non-empty `Content-Type` as the final `mediaType`, so CDN responses defaulting to `application/octet-stream` (or `text/plain`) skipped URL/byte-based detection and forwarded an unrecognized type into Claude Code's stream-json `media_type` field β Anthropic rejects those even when the bytes are a valid PNG/JPEG. The same flaw existed for base64 sources whose declared `mediaType` was generic. Introduce `pickImageMediaType(headerType, url, buffer)`: the header value is preferred only when it's a recognized `image/*` type we know how to extension- map; otherwise it falls through to URL extension hint β byte-signature sniff β raw header β `image/png` final fallback. Applied uniformly to URL fetch, URL cache hit, and base64 decode paths. Path sources are unchanged (their "header" is the file extension, which is already authoritative when present). **P2 β async spawnAgent rejections crashed the CLI** `spawnAgent` is now async and can reject during image normalization (missing local `--image` path, fetch failure, decode error). The CLI awaited it outside any try/catch, so user-input errors surfaced as unhandled rejections with stack traces instead of the friendly `log.error + process.exit` path used for prompt validation. Wrap the `await spawnAgent(...)` in try/catch, log the error message, exit 1 (matching the existing "Stream error from agent process" convention). **Tests** - `buildAgentInput.test.ts`: 3 new tests covering octet-stream URL Content-Type β byte sniff, octet-stream base64 declared type β byte sniff, generic header + URL extension hint preferred over header. - `hetero.test.ts`: 1 new test verifying spawnAgent rejection produces clean `exit(1)` instead of an unhandled rejection. Manually verified: `lh hetero exec --image /tmp/does-not-exist.png` β `[ERROR] Failed to start agent: ENOENT: no such file or directoryβ¦` + exit 1 Refs LOBE-8523. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
π» Change Type
π Related Issue
Refs LOBE-8523 (phase 1a follow-up; unblocks phase 1b ingest by making the shared input shape multimodal-ready).
Builds on #14431 (phase 1a).
π Description of Change
spawnAgentandlh hetero execpreviously accepted only a flat stringprompt, so attaching images required bypassing the shared package β which is exactly whatHeterogeneousAgentCtrdid. This PR unifies the input shape, lifts image handling into the shared layer, and exposes multimodal flags to the CLI.New shared input module (
packages/heterogeneous-agents/src/spawn/input/):types.tsβAgentPromptInput = string | AgentContentBlock[]; image sources accepturl(with optional id for cache dedupe), localpath, or inlinebase64.normalizeImage.tsβ fetch / read / decode β{ buffer, mediaType, path? }. Optional on-disk URL cache keyed bysha256(id || url). Byte-signature sniffing falls back when MIME is generic (application/octet-stream).buildAgentInput.tsβ single source of truth for per-agent serialization. Claude Code receives base64 image blocks inline in stream-json; Codex receives raw text on stdin + repeatable--image <path>flags.spawnAgentupgrade:prompt: AgentPromptInput(string is sugar for one text block).inputOptions?: { cacheDir, fetcher, imageMaterializeDir }forwarded tobuildAgentInput.CLI gets three input modes (
apps/cli/src/commands/hetero.ts):```bash
Terminal sugar β text + repeatable images
lh hetero exec --type claude-code
--prompt 'describe this'
--image ./screen.png
--image https://cdn.example/photo.jpg
--image data:image/png;base64,...
Programmatic / sandbox β full content-blocks JSON
lh hetero exec --type claude-code --input-json ./input.json
echo '[{"type":"text",...},{"type":"image","source":{...}}]' |
lh hetero exec --type claude-code --input-json -
Stdin auto-detect β JSON if first non-whitespace is { or [, else text
cat prompt.txt | lh hetero exec --type claude-code
```
--promptand--input-jsonare mutually exclusive;--imageis rejected with--input-json(the JSON should hold its own images).Desktop refactor:
HeterogeneousAgentCtrdrops ~100 lines of duplicated image cache / mime-sniffing / extension-guessing code. The two helpers (buildStreamJsonInput,resolveCliImagePaths) are now thin wrappers aroundbuildAgentInput/normalizeImage/materializeImageToPath. Driver interface and IPC contract are unchanged.π§ͺ How to Test
Automated:
bunx vitest run packages/heterogeneous-agentsβ 127/127 pass (10 newbuildAgentInputtests + 2 newspawnAgentmultimodal tests + 12 existing migrated to async API).bunx vitest run apps/cli/src/commands/hetero.test.tsβ 11/11 pass (4 new for--image/--input-json/ mutual exclusion; existing tests adapted to the now-asyncspawnAgent).bunx vitest run apps/desktop/src/main/controllers/__tests__/HeterogeneousAgentCtr.test.tsβ 28/28 pass; the path-traversal security tests retargeted to the sharednormalizeImage(same invariant, correct layer).bun run type-checkβ clean.End-to-end with real Claude Code:
```bash
tiny 73-byte red PNG fixture
bun apps/cli/src/index.ts hetero exec --type claude-code
--prompt "Reply with one sentence about the color."
--image /tmp/red.png
β 'I see a solid red color.' streamed as JSONL
JSON mode
bun apps/cli/src/index.ts hetero exec --type claude-code
--input-json /tmp/input.json
β 'I see a deep red/crimson color.'
```
Both runs produced the full event chain (stream_start β stream_chunk Γ N β step_complete Γ 2 β stream_end β agent_runtime_end), all stamped with the same auto-generated operationId, and Claude correctly identified the image color.
π Additional Information
Breaking change (internal):
spawnAgentis now async. The only caller in-tree waslh hetero exec; it is updated. Desktop'sHeterogeneousAgentCtrdoesn't usespawnAgentdirectly (host concerns differ) so it's unaffected.Why share the input module rather than just the CLI? Desktop was already doing the exact same image normalization manually, just below the shared layer. Lifting it removes the duplication, validates that the shared abstraction can express what desktop needs, and makes phase 1b's cloud-sandbox ingest path naturally multimodal β the server can pass content-blocks JSON straight through
lh hetero exec --input-json -without re-encoding.Why URL fetching in CLI? Cloud sandbox will pass URLs for user-attached images. Doing the fetch in shared (with disk cache) means the server doesn't need a pre-resolve step before invoking
lh hetero exec, and existing desktop URL caching semantics are preserved.π€ Generated with Claude Code