feat: enable enterprise web search for gemini models by hustxiayang · Pull Request #1526 · envoyproxy/ai-gateway

hustxiayang · 2025-11-10T17:21:00Z

Description

This is to enable feature Web Grounding for Enterprise: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/web-grounding-enterprise

Related Issues/PRs (if applicable)
#1417

codecov-commenter · 2025-11-10T19:15:07Z

Codecov Report

❌ Patch coverage is 73.33333% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.24%. Comparing base (03919d3) to head (947d4cc).

Files with missing lines	Patch %	Lines
internal/translator/gemini_helper.go	73.33%	3 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (73.33%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1526      +/-   ##
==========================================
- Coverage   83.26%   83.24%   -0.03%     
==========================================
  Files         137      137              
  Lines       12351    12364      +13     
==========================================
+ Hits        10284    10292       +8     
- Misses       1440     1445       +5     
  Partials      627      627

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

hustxiayang · 2025-11-20T18:11:32Z

/retest

hustxiayang · 2025-11-20T18:16:53Z

/retest

#1554) **Description** Some new features were introduced in gemini3: **1 thinking_level:** https://ai.google.dev/gemini-api/docs/gemini-3?thinking=low#thinking_level This is similar to reasoning_effort of openai, thus, unified them. **2 media_resolution** https://ai.google.dev/gemini-api/docs/gemini-3?thinking=low#media_resolution This is similar to detail in openai, thus, unified them. The difference is that openai does not provide a global config of media_resolution. Thus, added it as gcp specific, but still use detail to make the name consistent. **Some related PRs:** thinking_budget is in #1461 thinking_level and thinking_budget are both supported, but can not use them together. Other features under review: **1 web search:** #1526 **2 parse the thought summary:** #1521 --------- Signed-off-by: yxia216 <yxia216@bloomberg.net> Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> Signed-off-by: Sukumar Gaonkar <sgaonkar4@bloomberg.net> Co-authored-by: Alexa Griffith <agriffith50@bloomberg.net> Co-authored-by: Sukumar Gaonkar <sgaonkar4@bloomberg.net> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io>

mathetake · 2025-11-25T23:30:52Z

This needs a documentation update: https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/vendor-specific-fields (could you also backfill the doc for other fields added recently that lack docs if any?)

yuzisun · 2025-11-29T17:51:36Z

internal/apischema/openai/openai.go

+
+	// EnterpriseWebSearch controls whether to use Web Grounding for Enterprise
+	// https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/web-grounding-enterprise
+	EnterpriseWebSearch bool `json:"enterprise_search,omitzero"`


There is a list of supported tools, I think we should define a list here
https://github.com/googleapis/go-genai/blob/6a8184fcaf8bf15f0c566616a7b356560309be9b/types.go#L3934

@yuzisun you mean this list: https://github.com/googleapis/go-genai/blob/6a8184fcaf8bf15f0c566616a7b356560309be9b/types.go#L1406? Do we have plans to support them?

see how litellm supports the web search tool, I think we can extend the openai tool type?
https://docs.litellm.ai/docs/providers/vertex#enterprise-web-search

const ( ToolTypeFunction ToolType = "function" ToolTypeImageGeneration ToolType = "image_generation" ToolTypeEnterpriseWebSearch = "enterprise_search" ) type Tool struct { Type ToolType `json:"type"` Function *FunctionDefinition `json:"function,omitempty"` }

I am not against to it, but this is actually another example that whether we need to make these "ventor-specific" or just make the interface more "unified". I would follow it if others have no concerns on it.

I updated and followed your codes, but not litellm's. Users use tools=["type": "enterprise_search"]. I feel this is more natural.

Signed-off-by: yxia216 <yxia216@bloomberg.net>

hustxiayang · 2025-11-29T23:59:40Z

This needs a documentation update: https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/vendor-specific-fields (could you also backfill the doc for other fields added recently that lack docs if any?)

@mathetake Thanks a lot for your comment! @yuzisun started a pr #1590, so I left a comment about recent changes under that pr.

hustxiayang · 2025-11-30T00:28:52Z

/retest

hustxiayang · 2025-11-30T00:42:30Z

/retest

hustxiayang · 2025-11-30T01:01:38Z

/retest

Signed-off-by: yxia216 <yxia216@bloomberg.net>

nacx · 2025-12-01T12:20:23Z

I left a comment about recent changes under that pr.

Let's add the docs here to have this PR self-contained and rebase once that PR is merged.

yuzisun · 2025-12-05T02:08:34Z

I left a comment about recent changes under that pr.

Let's add the docs here to have this PR self-contained and rebase once that PR is merged.

We will open separate PR for doc, in general we are missing all the doc for tool calls.

yuzisun · 2025-12-05T02:12:44Z

internal/apischema/openai/openai.go

 	// https://cloud.google.com/vertex-ai/generative-ai/docs/reference/rest/v1/GenerateContentResponse#SafetyRating
 	SafetyRatings []*genai.SafetyRating `json:"safety_ratings,omitempty"`
+
+	// GroundingMetadata specifies sources used to ground generated content.


Can you add a reference link ?

thanks a lot! Added a link

Signed-off-by: yxia216 <yxia216@bloomberg.net>

envoyproxy#1554) **Description** Some new features were introduced in gemini3: **1 thinking_level:** https://ai.google.dev/gemini-api/docs/gemini-3?thinking=low#thinking_level This is similar to reasoning_effort of openai, thus, unified them. **2 media_resolution** https://ai.google.dev/gemini-api/docs/gemini-3?thinking=low#media_resolution This is similar to detail in openai, thus, unified them. The difference is that openai does not provide a global config of media_resolution. Thus, added it as gcp specific, but still use detail to make the name consistent. **Some related PRs:** thinking_budget is in envoyproxy#1461 thinking_level and thinking_budget are both supported, but can not use them together. Other features under review: **1 web search:** envoyproxy#1526 **2 parse the thought summary:** envoyproxy#1521 --------- Signed-off-by: yxia216 <yxia216@bloomberg.net> Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> Signed-off-by: Sukumar Gaonkar <sgaonkar4@bloomberg.net> Co-authored-by: Alexa Griffith <agriffith50@bloomberg.net> Co-authored-by: Sukumar Gaonkar <sgaonkar4@bloomberg.net> Co-authored-by: Ignasi Barrera <ignasi@tetrate.io> Signed-off-by: Erica Hughberg <erica.sundberg.90@gmail.com>

**Description** This is to enable feature `Web Grounding for Enterprise`: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/web-grounding-enterprise **Related Issues/PRs (if applicable)** envoyproxy#1417 --------- Signed-off-by: yxia216 <yxia216@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: Erica Hughberg <erica.sundberg.90@gmail.com>

…ni models (#1641) **Description** Adds support for Google Search grounding [1] as a tool type for Gemini models, complementing the existing enterprise web search support added in #1526. Enterprise search and Google Search are two different tools in Vertex. See [2] for more details. [1]: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/grounding-with-google-search [2]: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/web-grounding-enterprise#overview The implementation translates the google_search tool type to Gemini's GoogleSearch with support for all configuration options: - exclude_domains: Filter out specific domains from search results (Vertex AI only) - blocking_confidence: Set phishing block threshold (Vertex AI only) - time_range_filter: Restrict results to a time window (Gemini API only) Response grounding metadata is already handled by the existing GroundingMetadata field from #1526. **Related Issues/PRs (if applicable)** Related PR: #1526 **Special notes for reviewers (if applicable)** The config structs are prefixed with GCP (e.g., GCPGoogleSearchConfig, GCPTimeRangeFilter) to follow the existing pattern for vendor-specific extensions (GCPVertexAIVendorFields, GCPVertexAIGenerationConfig). --------- Signed-off-by: jamesbuddrige <jxbuddrige@hotmail.co.uk>

…ni models (envoyproxy#1641) **Description** Adds support for Google Search grounding [1] as a tool type for Gemini models, complementing the existing enterprise web search support added in envoyproxy#1526. Enterprise search and Google Search are two different tools in Vertex. See [2] for more details. [1]: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/grounding-with-google-search [2]: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/web-grounding-enterprise#overview The implementation translates the google_search tool type to Gemini's GoogleSearch with support for all configuration options: - exclude_domains: Filter out specific domains from search results (Vertex AI only) - blocking_confidence: Set phishing block threshold (Vertex AI only) - time_range_filter: Restrict results to a time window (Gemini API only) Response grounding metadata is already handled by the existing GroundingMetadata field from envoyproxy#1526. **Related Issues/PRs (if applicable)** Related PR: envoyproxy#1526 **Special notes for reviewers (if applicable)** The config structs are prefixed with GCP (e.g., GCPGoogleSearchConfig, GCPTimeRangeFilter) to follow the existing pattern for vendor-specific extensions (GCPVertexAIVendorFields, GCPVertexAIGenerationConfig). --------- Signed-off-by: jamesbuddrige <jxbuddrige@hotmail.co.uk>

…ni models (envoyproxy#1641) **Description** Adds support for Google Search grounding [1] as a tool type for Gemini models, complementing the existing enterprise web search support added in envoyproxy#1526. Enterprise search and Google Search are two different tools in Vertex. See [2] for more details. [1]: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/grounding-with-google-search [2]: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/grounding/web-grounding-enterprise#overview The implementation translates the google_search tool type to Gemini's GoogleSearch with support for all configuration options: - exclude_domains: Filter out specific domains from search results (Vertex AI only) - blocking_confidence: Set phishing block threshold (Vertex AI only) - time_range_filter: Restrict results to a time window (Gemini API only) Response grounding metadata is already handled by the existing GroundingMetadata field from envoyproxy#1526. **Related Issues/PRs (if applicable)** Related PR: envoyproxy#1526 **Special notes for reviewers (if applicable)** The config structs are prefixed with GCP (e.g., GCPGoogleSearchConfig, GCPTimeRangeFilter) to follow the existing pattern for vendor-specific extensions (GCPVertexAIVendorFields, GCPVertexAIGenerationConfig). --------- Signed-off-by: jamesbuddrige <jxbuddrige@hotmail.co.uk> Signed-off-by: yxia216 <yxia216@bloomberg.net>

hustxiayang requested a review from a team as a code owner November 10, 2025 17:21

dosubot bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Nov 10, 2025

hustxiayang marked this pull request as draft November 10, 2025 17:31

hustxiayang marked this pull request as ready for review November 10, 2025 19:09

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Nov 10, 2025

hustxiayang marked this pull request as draft November 10, 2025 19:12

hustxiayang force-pushed the gemini-search branch from 8f29ea8 to c395cb0 Compare November 10, 2025 19:12

hustxiayang marked this pull request as ready for review November 10, 2025 19:12

hustxiayang marked this pull request as draft November 10, 2025 19:15

hustxiayang mentioned this pull request Nov 20, 2025

feat: add new features(thinking_level and media_resolution) of gemini3 #1554

Merged

hustxiayang force-pushed the gemini-search branch from c395cb0 to da9bdfe Compare November 20, 2025 14:33

hustxiayang marked this pull request as ready for review November 20, 2025 16:22

hustxiayang marked this pull request as draft November 20, 2025 16:38

hustxiayang marked this pull request as ready for review November 21, 2025 04:15

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Nov 21, 2025

yuzisun reviewed Nov 29, 2025

View reviewed changes

hustxiayang force-pushed the gemini-search branch from 23d2ea3 to 8df3860 Compare November 29, 2025 23:57

hustxiayang added 6 commits November 29, 2025 18:57

init

dd63401

Signed-off-by: yxia216 <yxia216@bloomberg.net>

fix-test

441a464

Signed-off-by: yxia216 <yxia216@bloomberg.net>

fix-tests

a556bce

Signed-off-by: yxia216 <yxia216@bloomberg.net>

output

8c5fa53

Signed-off-by: yxia216 <yxia216@bloomberg.net>

fix-rebase

7db29b7

Signed-off-by: yxia216 <yxia216@bloomberg.net>

fix

1be159e

Signed-off-by: yxia216 <yxia216@bloomberg.net>

rebase

a451f7d

Signed-off-by: yxia216 <yxia216@bloomberg.net>

hustxiayang force-pushed the gemini-search branch from 8df3860 to a451f7d Compare November 29, 2025 23:57

hustxiayang added 2 commits November 29, 2025 21:14

Merge branch 'main' into gemini-search

435efc1

comments

5adce1f

Signed-off-by: yxia216 <yxia216@bloomberg.net>

Merge branch 'main' into gemini-search

dc913bb

yuzisun reviewed Dec 5, 2025

View reviewed changes

fix-comments

28fb3d1

Signed-off-by: yxia216 <yxia216@bloomberg.net>

yuzisun approved these changes Dec 6, 2025

View reviewed changes

Merge branch 'main' into gemini-search

947d4cc

yuzisun merged commit d78bbb9 into envoyproxy:main Dec 6, 2025
30 checks passed

jamesbuddrige mentioned this pull request Dec 9, 2025

feat: add support for Google Search grounding as a tool type for Gemini models #1641

Merged

Conversation

hustxiayang commented Nov 10, 2025

Uh oh!

codecov-commenter commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hustxiayang commented Nov 20, 2025

Uh oh!

hustxiayang commented Nov 20, 2025

Uh oh!

mathetake commented Nov 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hustxiayang commented Nov 29, 2025

Uh oh!

hustxiayang commented Nov 30, 2025

Uh oh!

hustxiayang commented Nov 30, 2025

Uh oh!

hustxiayang commented Nov 30, 2025

Uh oh!

nacx commented Dec 1, 2025

Uh oh!

yuzisun commented Dec 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Nov 10, 2025 •

edited

Loading