β¨ feat: add visual understanding tool#14376
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codecov Reportβ
All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## canary #14376 +/- ##
===========================================
+ Coverage 68.96% 84.89% +15.92%
===========================================
Files 2403 589 -1814
Lines 209462 42088 -167374
Branches 26268 6441 -19827
===========================================
- Hits 144465 35732 -108733
+ Misses 64854 6213 -58641
Partials 143 143
Flags with carried forward coverage won't be shown. Click here to find out more.
π New features to boost your workflow:
|
There was a problem hiding this comment.
π‘ Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f6f8f9a449
βΉοΈ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with π.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const [sourceMessage] = await messageModel.queryByIds([this.messageId], { | ||
| postProcessUrl: (path) => fileService.getFullFileUrl(path), |
There was a problem hiding this comment.
Resolve visual source message from user turn
This runtime assumes context.messageId is the user turn with attachments, but in resume/continue flows the parent message can be an assistant message, so querying only this.messageId can return no imageList/videoList and produce NO_VISUAL_FILES even when the preceding user turn has visuals. That breaks visual-fallback tool calls during regeneration/continuation for non-vision models; the runtime should walk to the related user message (as the client executor does) instead of reading only one message id.
Useful? React with πΒ / π.
| const fallbackSupportVision = useModelSupportVision( | ||
| visualUnderstanding?.model ?? '', | ||
| visualUnderstanding?.provider ?? '', | ||
| ); | ||
| const fallbackSupportVideo = useModelSupportVideo( | ||
| visualUnderstanding?.model ?? '', | ||
| visualUnderstanding?.provider ?? '', |
There was a problem hiding this comment.
Do not block fallback uploads on unknown visual model cards
The fallback gate is derived from useModelSupportVision/useModelSupportVideo for the configured visualUnderstanding model, but those selectors return false when the model is not in enabledAiModels. In that case, uploads are rejected client-side even though server-side visual understanding is enabled by VISUAL_UNDERSTANDING_* envs and runtime execution can still proceed, so custom/hidden fallback model IDs become unusable from the UI.
Useful? React with πΒ / π.
π» Change Type
π Related Issue
Related to LOBE-8387
π Description of Change
π§ͺ How to Test
Commands:
πΈ Screenshots / Videos
N/A
π Additional Information
Security review: checked the submodule diff for cloud/business/billing-specific implementation details; no new cloud-specific logic is exposed.