✨ feat: add auto aspect ratio and image search support for Nano Banana 2#12537
Merged
Conversation
|
@sxjeru is attempting to deploy a commit to the LobeHub OSS Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
Reviewer's GuideAdds support for auto aspect ratio for Nano Banana image models and integrates Google image search results into grounding metadata and search tools, updating Google/Vertex streaming transforms, search tooling, and tests accordingly. Sequence diagram for GoogleGenerativeAIStream grounding with image searchsequenceDiagram
participant GoogleAPIStream
participant GoogleGenerativeAIStream
participant transformGoogleGenerativeAIStream
participant SSEClient
GoogleAPIStream->>GoogleGenerativeAIStream: ReadableStream<Chunk>
loop for each chunk
GoogleGenerativeAIStream->>transformGoogleGenerativeAIStream: candidate.groundingMetadata
activate transformGoogleGenerativeAIStream
transformGoogleGenerativeAIStream->>transformGoogleGenerativeAIStream: Extract groundingChunks, webSearchQueries, imageSearchQueries
transformGoogleGenerativeAIStream->>transformGoogleGenerativeAIStream: Split webChunks = filter(web)
transformGoogleGenerativeAIStream->>transformGoogleGenerativeAIStream: Split imageChunks = filter(image)
alt has webChunks
transformGoogleGenerativeAIStream->>SSEClient: event grounding
Note over transformGoogleGenerativeAIStream,SSEClient: data.citations = map(webChunks)
end
alt has imageChunks
transformGoogleGenerativeAIStream->>SSEClient: event grounding
Note over transformGoogleGenerativeAIStream,SSEClient: data.imageResults = map(imageChunks)
end
alt has imageSearchQueries
transformGoogleGenerativeAIStream->>SSEClient: event grounding
Note over transformGoogleGenerativeAIStream,SSEClient: data.imageSearchQueries
end
transformGoogleGenerativeAIStream-->>GoogleGenerativeAIStream: text + grounding events
deactivate transformGoogleGenerativeAIStream
end
GoogleGenerativeAIStream-->>SSEClient: usage + stop events
Sequence diagram for VertexAIStream grounding with image searchsequenceDiagram
participant VertexAPIStream
participant VertexAIStream
participant transformVertexAIStream
participant SSEClient
VertexAPIStream->>VertexAIStream: ReadableStream<Chunk>
loop for each chunk
VertexAIStream->>transformVertexAIStream: candidate.groundingMetadata
activate transformVertexAIStream
transformVertexAIStream->>transformVertexAIStream: Extract groundingChunks, webSearchQueries, imageSearchQueries
transformVertexAIStream->>transformVertexAIStream: Split webChunks = filter(web)
transformVertexAIStream->>transformVertexAIStream: Split imageChunks = filter(image)
alt has webChunks
transformVertexAIStream->>SSEClient: event grounding with citations
end
alt has imageChunks
transformVertexAIStream->>SSEClient: event grounding with imageResults
end
alt has imageSearchQueries
transformVertexAIStream->>SSEClient: event grounding with imageSearchQueries
end
transformVertexAIStream-->>VertexAIStream: text + grounding events
deactivate transformVertexAIStream
end
VertexAIStream-->>SSEClient: usage + stop events
Class diagram for updated GroundingSearch and image citation typesclassDiagram
class CitationItem {
+string favicon
+string title
+string url
}
class ImageCitationItem {
+string domain
+string imageUri
+string sourceUri
+string title
}
class GroundingSearch {
+CitationItem[] citations
+ImageCitationItem[] imageResults
+string[] imageSearchQueries
+string[] searchQueries
}
class GoogleGenerativeAIStream {
+ReadableStream rawStream
+GoogleGenerativeAIStream(rawStream)
}
class VertexAIStream {
+ReadableStream rawStream
+VertexAIStream(rawStream)
}
class LobeGoogleAI {
+Set modelsWithModalities
+Set modelsWithImageSearch
+chat(payload)
+buildGoogleToolsWithSearch(tools)
}
GroundingSearch "0..*" --> CitationItem : uses
GroundingSearch "0..*" --> ImageCitationItem : uses
GoogleGenerativeAIStream --> GroundingSearch : emits_grounding_events
VertexAIStream --> GroundingSearch : emits_grounding_events
LobeGoogleAI --> GroundingSearch : parses_grounding_metadata
Flow diagram for building Google search tools with optional image searchflowchart TD
A_start[Start buildGoogleToolsWithSearch] --> B_hasSearch{hasSearch?}
B_hasSearch -- No --> Z_returnTools[Return function/urlContext tools only]
B_hasSearch -- Yes --> C_hasUrlContext{hasUrlContext?}
C_hasUrlContext -- Yes --> D_modelSupportsImageSearch{model in modelsWithImageSearch?}
C_hasUrlContext -- No --> G_modelSupportsImageSearch2{model in modelsWithImageSearch?}
D_modelSupportsImageSearch -- Yes --> E_toolsUrlAndImageSearch["tools = [{ urlContext: {} }, { googleSearch: { searchTypes: { imageSearch: {}, webSearch: {} } } }]"]
D_modelSupportsImageSearch -- No --> F_toolsUrlAndWebSearch["tools = [{ urlContext: {} }, { googleSearch: {} }]"]
G_modelSupportsImageSearch2 -- Yes --> H_toolsImageSearchOnly["tools = [{ googleSearch: { searchTypes: { imageSearch: {}, webSearch: {} } } }]"]
G_modelSupportsImageSearch2 -- No --> I_toolsWebSearchOnly["tools = [{ googleSearch: {} }]"]
E_toolsUrlAndImageSearch --> J_done[Return tools]
F_toolsUrlAndWebSearch --> J_done
H_toolsImageSearchOnly --> J_done
I_toolsWebSearchOnly --> J_done
Z_returnTools --> J_done
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The grounding metadata mapping logic for Google and Vertex streams is now effectively duplicated (splitting web vs image chunks and building
GroundingSearch), which might be worth extracting into a shared helper to ensure behavior stays in sync across providers. - The
NANO_BANANA_ASPECT_RATIOS/NANO_BANANA_2_ASPECT_RATIOSdefinitions (including the newautooption and defaults) are now maintained in multiple places (model-bankand UI controls); consider centralizing these constants to reduce the risk of future divergence between runtime and UI.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The grounding metadata mapping logic for Google and Vertex streams is now effectively duplicated (splitting web vs image chunks and building `GroundingSearch`), which might be worth extracting into a shared helper to ensure behavior stays in sync across providers.
- The `NANO_BANANA_ASPECT_RATIOS`/`NANO_BANANA_2_ASPECT_RATIOS` definitions (including the new `auto` option and defaults) are now maintained in multiple places (`model-bank` and UI controls); consider centralizing these constants to reduce the risk of future divergence between runtime and UI.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## canary #12537 +/- ##
==========================================
+ Coverage 73.47% 73.50% +0.02%
==========================================
Files 1389 1392 +3
Lines 113570 113772 +202
Branches 14691 12904 -1787
==========================================
+ Hits 83441 83623 +182
- Misses 30024 30041 +17
- Partials 105 108 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Nano Banana (Gemini image-capable) configuration to support an auto aspect ratio (by omitting the provider aspectRatio parameter) and adds preliminary plumbing for Google image search tooling and image-search grounding parsing.
Changes:
- Add
autoto Nano Banana/Nano Banana 2 aspect-ratio enums and switch defaults toautoin UI + model parameter definitions. - In Google image-capable calls, omit
imageConfig.aspectRatiowhen the selected ratio isauto. - Add image-search related types/schemas and parse image grounding chunks into
imageResults/imageSearchQueries; enablegoogleSearch.searchTypes.imageSearchfor the Nano Banana 2 chat model; bump@google/genai.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/features/ModelSwitchPanel/components/ControlsForm/ImageAspectRatioSelect.tsx | Adds auto option and makes it the default for Nano Banana aspect ratio control. |
| src/features/ModelSwitchPanel/components/ControlsForm/ImageAspectRatio2Select.tsx | Adds auto option and makes it the default for Nano Banana 2 aspect ratio control. |
| packages/types/src/search.ts | Extends grounding/search types with image search result structures + zod schemas. |
| packages/model-runtime/src/providers/google/index.ts | Omits imageConfig on auto and enables imageSearch tool config for supported model(s). |
| packages/model-runtime/src/providers/google/index.test.ts | Adds coverage for the new imageSearch tool configuration. |
| packages/model-runtime/src/providers/google/createImage.ts | Omits imageConfig when aspect ratio is auto for image generation. |
| packages/model-runtime/src/providers/google/createImage.test.ts | Tests that auto does not send imageConfig. |
| packages/model-runtime/src/core/streams/vertex-ai.ts | Parses image grounding chunks into imageResults/imageSearchQueries. |
| packages/model-runtime/src/core/streams/vertex-ai.test.ts | Adds tests for image grounding parsing in VertexAI stream. |
| packages/model-runtime/src/core/streams/google/index.ts | Parses image grounding chunks into imageResults/imageSearchQueries. |
| packages/model-runtime/src/core/streams/google/google-ai.test.ts | Adds tests for image grounding parsing in Google stream. |
| packages/model-runtime/package.json | Bumps @google/genai dependency version. |
| packages/model-bank/src/aiModels/lobehub/utils.ts | Adds auto to Nano Banana aspect ratios and sets default to auto. |
| packages/model-bank/src/aiModels/google.ts | Adds auto to Nano Banana aspect ratios and sets default to auto for Google model cards. |
Comments suppressed due to low confidence (1)
packages/model-runtime/src/providers/google/index.ts:179
- The chat path now treats
imageAspectRatio === 'auto'as “omit imageConfig”. There’s test coverage for tool building inindex.test.ts, but no test asserting the generatedGenerateContentConfig.imageConfigbehavior for'auto'vs a concrete ratio inLobeGoogleAI.chat. Adding a unit test here would help prevent regressions in how aspect ratio is forwarded to Gemini.
const config: GenerateContentConfig = {
abortSignal: originalSignal,
imageConfig:
modelsWithModalities.has(model) && imageAspectRatio && imageAspectRatio !== 'auto'
? {
aspectRatio: imageAspectRatio,
imageSize: imageResolution,
}
: undefined,
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a545ff646a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Contributor
Author
Further improvement to make website logos display correctly.
Make tool call (Google search) token consumption reflected.
Original Content进一步改进，使网站 logo 正常显示。
使工具调用（google 搜索）token 消耗能被体现。
|
Contributor
Author
|
Image search results can now be displayed. 
Original Content搜图结果已可显示。
|
Contributor
Author
|
Implement parsing of search result references in the body text, such as image_0.png, etc. 
Original Content实现解析正文中的如 image_0.png 等搜索结果引用。
|
…and update favicon handling in SearchGrounding component
…stency in test cases
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Member
|
❤️ Great PR @sxjeru ❤️ The growth of project is inseparable from user feedback and contribution, thanks for your contribution! If you are interesting with the lobehub developer community, please join our discord and then dm @arvinxx or @canisminor1990. They will invite you to our private developer channel. We are talking about the lobe-chat development or sharing ai newsletter around the world. |
arvinxx
pushed a commit
that referenced
this pull request
Mar 4, 2026
…a 2 (#12537) * Update sync.yml * ✨ feat: update aspect ratio defaults to 'auto' for image generation models * ✨ feat: enhance grounding metadata handling with image search results support * ✨ feat: filter empty strings from searchQueries in groundingMetadata and update favicon handling in SearchGrounding component * ✨ feat: add inputToolTokens tracking and update related components for tool usage * ✨ feat: enhance search grounding with image results and update related components * ✨ feat: add ImageSearchRef component and related tests for image reference handling * fix test: rename VertexAIStream to GoogleGenerativeAIStream for consistency in test cases * Update sync.yml
MarioJames
pushed a commit
to MarioJames/lobehub
that referenced
this pull request
Mar 9, 2026
…a 2 (lobehub#12537) * Update sync.yml * ✨ feat: update aspect ratio defaults to 'auto' for image generation models * ✨ feat: enhance grounding metadata handling with image search results support * ✨ feat: filter empty strings from searchQueries in groundingMetadata and update favicon handling in SearchGrounding component * ✨ feat: add inputToolTokens tracking and update related components for tool usage * ✨ feat: enhance search grounding with image results and update related components * ✨ feat: add ImageSearchRef component and related tests for image reference handling * fix test: rename VertexAIStream to GoogleGenerativeAIStream for consistency in test cases * Update sync.yml
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💻 Change Type
🔗 Related Issue
🔀 Description of Change
宽高比添加 auto 项，以留空参数生成和提供图片尺寸匹配的图。
尝试支持图片搜索。（3.1 Flash 引入）
https://ai.google.dev/gemini-api/docs/image-generation#image-search
确认可用。预览图在下面。
🧪 How to Test
📸 Screenshots / Videos
📝 Additional Information
Summary by Sourcery
Support auto image aspect ratio for Nano Banana image models and extend Google Gemini/Vertex search integration with image search results.
New Features:
Enhancements:
Tests: