refactor: use Envoy native router by mathetake · Pull Request #793 · envoyproxy/ai-gateway

mathetake · 2025-06-28T19:11:13Z

Description

This commit removes the handwritten header matching code from the extproc, and instead starts utilizing the hardened envoy native router.

Historically, we had only one giant extproc filter where we did all logics including model name extraction, routing and then body transformation & upstream authorization. Since #599, we split into two external processor filters; one sits at the normal HTTP router and the other is configured at the per-cluster upstream HTTP filter. In theory, the one at HTTP router has only one job on request path: extracting model name from the request body. However, due to the historical reason, the handwritten router logic component remained, and that comes with not only a maintenance cost (forcing a complex extproc & control plane orchestration) but also a potential security vulnerability. In fact, writing header matching logic can be an easy attack surface, so if it's possible, we should avoid writing our own header matching (routing logic) but should rely on the battle-tested hardened envoy native router.

With this commit, now a regex matching is available as well as there's no difference between HTTPRoute's matching and AIGatewayRoute's matching implementation. This also opens up a possibility to support path matching in our rule.

Related Issues/PRs (if applicable)

Ref #612
Ref #73

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> # Conflicts: # api/v1alpha1/api.go # internal/controller/ai_gateway_route.go # internal/extproc/embeddings_processor.go

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

Xunzhuo

love the simplification! let us get this in

Xunzhuo · 2025-07-04T03:18:09Z

It looks like a conflict @mathetake can resolve it?

# Conflicts: # tests/crdcel/main_test.go

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

Xunzhuo · 2025-07-04T03:46:49Z

@mathetake feel free to merge it after CI passed

mathetake · 2025-07-04T03:56:54Z

Ok looks like I need to add more coverage

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

**Description** Before #793, the case where no matching route found was handled in the extproc and the 404 immediate response was returned from there, but after that, it naturally results in the "unreachable" default route and swallowed the indication of no matching and it made it impossible to reason about the 500 error on that case. In other words, this fixes the regression in #793 to return the proper 404 response. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

**Description** Before envoyproxy#793, the case where no matching route found was handled in the extproc and the 404 immediate response was returned from there, but after that, it naturally results in the "unreachable" default route and swallowed the indication of no matching and it made it impossible to reason about the 500 error on that case. In other words, this fixes the regression in envoyproxy#793 to return the proper 404 response. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

@mathetake

**Description** This commit removes the handwritten header matching code from the extproc, and instead starts utilizing the hardened envoy native router. Historically, we had only one giant extproc filter where we did all logics including model name extraction, routing and then body transformation & upstream authorization. Since envoyproxy#599, we split into two external processor filters; one sits at the normal HTTP router and the other is configured at the per-cluster upstream HTTP filter. In theory, the one at HTTP router has only one job on request path: extracting model name from the request body. However, due to the historical reason, the handwritten router logic component remained, and that comes with not only a maintenance cost (forcing a complex extproc & control plane orchestration) but also a potential security vulnerability. In fact, writing header matching logic can be an easy attack surface, so if it's possible, we should avoid writing our own header matching (routing logic) but should rely on the battle-tested hardened envoy native router. With this commit, now a regex matching is available as well as there's no difference between HTTPRoute's matching and AIGatewayRoute's matching implementation. This also opens up a possibility to support path matching in our rule. **Related Issues/PRs (if applicable)** Ref envoyproxy#612 Ref envoyproxy#73 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: fix aigw parentRefs in fallback (envoyproxy#824) **Description** This PR fixed the AIGatewayRoute parentRefs in fallback guides. Signed-off-by: bitliu <bitliu@tencent.com> chore: make test-e2e logs visible (envoyproxy#825) **Description** This PR is to make test-e2e logs visible in local. Signed-off-by: bitliu <bitliu@tencent.com> extproc: account for parallel tool calls (envoyproxy#813) **Description** Resolves envoyproxy#736 Assistant that calls multiple tools are expected to group tool result in the same message. Adding logic for that! --------- Signed-off-by: Aaron Choo <achoo30@bloomberg.net> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net> build(deps): bump google.golang.org/genai from 1.13.0 to 1.14.0 (envoyproxy#833) extproc: return 404 instead of 500 for unknown path (envoyproxy#835) **Description** Previously, unknown path was responded as an internal error as opposed to the fact that it's an 404 with the user input root cause. This fixes the extproc code that way, now that users will be able to know what's wrong with the operation instead of getting the cryptic 500 error. **Related Issues/PRs (if applicable)** Contributes to envoyproxy#810 Closes envoyproxy#724 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: add endpoint support (envoyproxy#787) **Description** This PR adds the endpoint support pages for EAGW. **Related Issues/PRs (if applicable)** Fixes: envoyproxy#705 **Special notes for reviewers (if applicable)** @mathetake --------- Signed-off-by: bitliu <bitliu@tencent.com> Co-authored-by: Erica Hughberg <erica.sundberg.90@gmail.com> controller: return 404 instead of 500 for no matching (envoyproxy#837) **Description** Before envoyproxy#793, the case where no matching route found was handled in the extproc and the 404 immediate response was returned from there, but after that, it naturally results in the "unreachable" default route and swallowed the indication of no matching and it made it impossible to reason about the 500 error on that case. In other words, this fixes the regression in envoyproxy#793 to return the proper 404 response. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> update Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> precommit passing Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> remove header hotfix Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> precommit working Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more test coverage Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more test coverage Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> test: adds real provider embeddings test & update doc (envoyproxy#841) **Description** This adds embeddings endpoint tests with the providers that support the endpoint. This only added the providers for which we have credentials. According to the testing situation we have right now, this also clarifies in the "Supported Endpoints" page that which provider is tested and which is not for each endpoint. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> cli: adds default route test (envoyproxy#842) **Description** This adds an additional test to aigw run command so that we can verify that setting the default route is possible. **Related Issues/PRs (if applicable)** Closes envoyproxy#612 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> build(deps): bump helm.sh/helm/v3 from 3.17.3 to 3.18.4 (envoyproxy#845) test: fixes TestStartConfigWatcher flake (envoyproxy#843) controller: ensure eg rollout when deployed as daemonset (envoyproxy#831) **Description** This PR handles the rollout for envoy gateway during ai gateway extproc upgrade when deployed as daemonset. Related Issues/PRs (if applicable) Related PR: envoyproxy#699 --------- Signed-off-by: Dan Sun <dsun20@bloomberg.net> make test var Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net>

@mathetake

Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> lint no err Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add translation Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> update so tests work Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more tests Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> remove print Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> refactor: deprecate targetRefs in favor or parentRefs (envoyproxy#821) docs: add epp integration proposal (envoyproxy#771) **Description** This PR adds the proposal for supporting Integration with Endpoint Picker(GIE) Related to envoyproxy#423 --------- Signed-off-by: bitliu <bitliu@tencent.com> Co-authored-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: update epp outdated logics (envoyproxy#822) refactor: use Envoy native router (envoyproxy#793) **Description** This commit removes the handwritten header matching code from the extproc, and instead starts utilizing the hardened envoy native router. Historically, we had only one giant extproc filter where we did all logics including model name extraction, routing and then body transformation & upstream authorization. Since envoyproxy#599, we split into two external processor filters; one sits at the normal HTTP router and the other is configured at the per-cluster upstream HTTP filter. In theory, the one at HTTP router has only one job on request path: extracting model name from the request body. However, due to the historical reason, the handwritten router logic component remained, and that comes with not only a maintenance cost (forcing a complex extproc & control plane orchestration) but also a potential security vulnerability. In fact, writing header matching logic can be an easy attack surface, so if it's possible, we should avoid writing our own header matching (routing logic) but should rely on the battle-tested hardened envoy native router. With this commit, now a regex matching is available as well as there's no difference between HTTPRoute's matching and AIGatewayRoute's matching implementation. This also opens up a possibility to support path matching in our rule. **Related Issues/PRs (if applicable)** Ref envoyproxy#612 Ref envoyproxy#73 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: fix aigw parentRefs in fallback (envoyproxy#824) **Description** This PR fixed the AIGatewayRoute parentRefs in fallback guides. Signed-off-by: bitliu <bitliu@tencent.com> chore: make test-e2e logs visible (envoyproxy#825) **Description** This PR is to make test-e2e logs visible in local. Signed-off-by: bitliu <bitliu@tencent.com> extproc: account for parallel tool calls (envoyproxy#813) **Description** Resolves envoyproxy#736 Assistant that calls multiple tools are expected to group tool result in the same message. Adding logic for that! --------- Signed-off-by: Aaron Choo <achoo30@bloomberg.net> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net> build(deps): bump google.golang.org/genai from 1.13.0 to 1.14.0 (envoyproxy#833) extproc: return 404 instead of 500 for unknown path (envoyproxy#835) **Description** Previously, unknown path was responded as an internal error as opposed to the fact that it's an 404 with the user input root cause. This fixes the extproc code that way, now that users will be able to know what's wrong with the operation instead of getting the cryptic 500 error. **Related Issues/PRs (if applicable)** Contributes to envoyproxy#810 Closes envoyproxy#724 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: add endpoint support (envoyproxy#787) **Description** This PR adds the endpoint support pages for EAGW. **Related Issues/PRs (if applicable)** Fixes: envoyproxy#705 **Special notes for reviewers (if applicable)** @mathetake --------- Signed-off-by: bitliu <bitliu@tencent.com> Co-authored-by: Erica Hughberg <erica.sundberg.90@gmail.com> controller: return 404 instead of 500 for no matching (envoyproxy#837) **Description** Before envoyproxy#793, the case where no matching route found was handled in the extproc and the 404 immediate response was returned from there, but after that, it naturally results in the "unreachable" default route and swallowed the indication of no matching and it made it impossible to reason about the 500 error on that case. In other words, this fixes the regression in envoyproxy#793 to return the proper 404 response. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> update Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> precommit passing Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> remove header hotfix Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> precommit working Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more test coverage Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more test coverage Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> test: adds real provider embeddings test & update doc (envoyproxy#841) **Description** This adds embeddings endpoint tests with the providers that support the endpoint. This only added the providers for which we have credentials. According to the testing situation we have right now, this also clarifies in the "Supported Endpoints" page that which provider is tested and which is not for each endpoint. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> cli: adds default route test (envoyproxy#842) **Description** This adds an additional test to aigw run command so that we can verify that setting the default route is possible. **Related Issues/PRs (if applicable)** Closes envoyproxy#612 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> build(deps): bump helm.sh/helm/v3 from 3.17.3 to 3.18.4 (envoyproxy#845) test: fixes TestStartConfigWatcher flake (envoyproxy#843) controller: ensure eg rollout when deployed as daemonset (envoyproxy#831) **Description** This PR handles the rollout for envoy gateway during ai gateway extproc upgrade when deployed as daemonset. Related Issues/PRs (if applicable) Related PR: envoyproxy#699 --------- Signed-off-by: Dan Sun <dsun20@bloomberg.net> make test var Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net>

@mathetake

Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> update paralleltoolcalls Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add back system helper Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> lint no err Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add translation Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> update so tests work Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more tests Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> remove print Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> refactor: deprecate targetRefs in favor or parentRefs (envoyproxy#821) docs: add epp integration proposal (envoyproxy#771) **Description** This PR adds the proposal for supporting Integration with Endpoint Picker(GIE) Related to envoyproxy#423 --------- Signed-off-by: bitliu <bitliu@tencent.com> Co-authored-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: update epp outdated logics (envoyproxy#822) refactor: use Envoy native router (envoyproxy#793) **Description** This commit removes the handwritten header matching code from the extproc, and instead starts utilizing the hardened envoy native router. Historically, we had only one giant extproc filter where we did all logics including model name extraction, routing and then body transformation & upstream authorization. Since envoyproxy#599, we split into two external processor filters; one sits at the normal HTTP router and the other is configured at the per-cluster upstream HTTP filter. In theory, the one at HTTP router has only one job on request path: extracting model name from the request body. However, due to the historical reason, the handwritten router logic component remained, and that comes with not only a maintenance cost (forcing a complex extproc & control plane orchestration) but also a potential security vulnerability. In fact, writing header matching logic can be an easy attack surface, so if it's possible, we should avoid writing our own header matching (routing logic) but should rely on the battle-tested hardened envoy native router. With this commit, now a regex matching is available as well as there's no difference between HTTPRoute's matching and AIGatewayRoute's matching implementation. This also opens up a possibility to support path matching in our rule. **Related Issues/PRs (if applicable)** Ref envoyproxy#612 Ref envoyproxy#73 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: fix aigw parentRefs in fallback (envoyproxy#824) **Description** This PR fixed the AIGatewayRoute parentRefs in fallback guides. Signed-off-by: bitliu <bitliu@tencent.com> chore: make test-e2e logs visible (envoyproxy#825) **Description** This PR is to make test-e2e logs visible in local. Signed-off-by: bitliu <bitliu@tencent.com> extproc: account for parallel tool calls (envoyproxy#813) **Description** Resolves envoyproxy#736 Assistant that calls multiple tools are expected to group tool result in the same message. Adding logic for that! --------- Signed-off-by: Aaron Choo <achoo30@bloomberg.net> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net> build(deps): bump google.golang.org/genai from 1.13.0 to 1.14.0 (envoyproxy#833) extproc: return 404 instead of 500 for unknown path (envoyproxy#835) **Description** Previously, unknown path was responded as an internal error as opposed to the fact that it's an 404 with the user input root cause. This fixes the extproc code that way, now that users will be able to know what's wrong with the operation instead of getting the cryptic 500 error. **Related Issues/PRs (if applicable)** Contributes to envoyproxy#810 Closes envoyproxy#724 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: add endpoint support (envoyproxy#787) **Description** This PR adds the endpoint support pages for EAGW. **Related Issues/PRs (if applicable)** Fixes: envoyproxy#705 **Special notes for reviewers (if applicable)** @mathetake --------- Signed-off-by: bitliu <bitliu@tencent.com> Co-authored-by: Erica Hughberg <erica.sundberg.90@gmail.com> controller: return 404 instead of 500 for no matching (envoyproxy#837) **Description** Before envoyproxy#793, the case where no matching route found was handled in the extproc and the 404 immediate response was returned from there, but after that, it naturally results in the "unreachable" default route and swallowed the indication of no matching and it made it impossible to reason about the 500 error on that case. In other words, this fixes the regression in envoyproxy#793 to return the proper 404 response. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> update Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> precommit passing Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> remove header hotfix Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> precommit working Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more test coverage Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more test coverage Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> test: adds real provider embeddings test & update doc (envoyproxy#841) **Description** This adds embeddings endpoint tests with the providers that support the endpoint. This only added the providers for which we have credentials. According to the testing situation we have right now, this also clarifies in the "Supported Endpoints" page that which provider is tested and which is not for each endpoint. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> cli: adds default route test (envoyproxy#842) **Description** This adds an additional test to aigw run command so that we can verify that setting the default route is possible. **Related Issues/PRs (if applicable)** Closes envoyproxy#612 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> build(deps): bump helm.sh/helm/v3 from 3.17.3 to 3.18.4 (envoyproxy#845) test: fixes TestStartConfigWatcher flake (envoyproxy#843) controller: ensure eg rollout when deployed as daemonset (envoyproxy#831) **Description** This PR handles the rollout for envoy gateway during ai gateway extproc upgrade when deployed as daemonset. Related Issues/PRs (if applicable) Related PR: envoyproxy#699 --------- Signed-off-by: Dan Sun <dsun20@bloomberg.net> make test var Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net>

@mathetake

Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> update paralleltoolcalls Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add back system helper Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> lint no err Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add translation Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> update so tests work Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more tests Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> remove print Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> refactor: deprecate targetRefs in favor or parentRefs (envoyproxy#821) docs: add epp integration proposal (envoyproxy#771) **Description** This PR adds the proposal for supporting Integration with Endpoint Picker(GIE) Related to envoyproxy#423 --------- Signed-off-by: bitliu <bitliu@tencent.com> Co-authored-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: update epp outdated logics (envoyproxy#822) refactor: use Envoy native router (envoyproxy#793) **Description** This commit removes the handwritten header matching code from the extproc, and instead starts utilizing the hardened envoy native router. Historically, we had only one giant extproc filter where we did all logics including model name extraction, routing and then body transformation & upstream authorization. Since envoyproxy#599, we split into two external processor filters; one sits at the normal HTTP router and the other is configured at the per-cluster upstream HTTP filter. In theory, the one at HTTP router has only one job on request path: extracting model name from the request body. However, due to the historical reason, the handwritten router logic component remained, and that comes with not only a maintenance cost (forcing a complex extproc & control plane orchestration) but also a potential security vulnerability. In fact, writing header matching logic can be an easy attack surface, so if it's possible, we should avoid writing our own header matching (routing logic) but should rely on the battle-tested hardened envoy native router. With this commit, now a regex matching is available as well as there's no difference between HTTPRoute's matching and AIGatewayRoute's matching implementation. This also opens up a possibility to support path matching in our rule. **Related Issues/PRs (if applicable)** Ref envoyproxy#612 Ref envoyproxy#73 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: fix aigw parentRefs in fallback (envoyproxy#824) **Description** This PR fixed the AIGatewayRoute parentRefs in fallback guides. Signed-off-by: bitliu <bitliu@tencent.com> chore: make test-e2e logs visible (envoyproxy#825) **Description** This PR is to make test-e2e logs visible in local. Signed-off-by: bitliu <bitliu@tencent.com> extproc: account for parallel tool calls (envoyproxy#813) **Description** Resolves envoyproxy#736 Assistant that calls multiple tools are expected to group tool result in the same message. Adding logic for that! --------- Signed-off-by: Aaron Choo <achoo30@bloomberg.net> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net> build(deps): bump google.golang.org/genai from 1.13.0 to 1.14.0 (envoyproxy#833) extproc: return 404 instead of 500 for unknown path (envoyproxy#835) **Description** Previously, unknown path was responded as an internal error as opposed to the fact that it's an 404 with the user input root cause. This fixes the extproc code that way, now that users will be able to know what's wrong with the operation instead of getting the cryptic 500 error. **Related Issues/PRs (if applicable)** Contributes to envoyproxy#810 Closes envoyproxy#724 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> docs: add endpoint support (envoyproxy#787) **Description** This PR adds the endpoint support pages for EAGW. **Related Issues/PRs (if applicable)** Fixes: envoyproxy#705 **Special notes for reviewers (if applicable)** @mathetake --------- Signed-off-by: bitliu <bitliu@tencent.com> Co-authored-by: Erica Hughberg <erica.sundberg.90@gmail.com> controller: return 404 instead of 500 for no matching (envoyproxy#837) **Description** Before envoyproxy#793, the case where no matching route found was handled in the extproc and the 404 immediate response was returned from there, but after that, it naturally results in the "unreachable" default route and swallowed the indication of no matching and it made it impossible to reason about the 500 error on that case. In other words, this fixes the regression in envoyproxy#793 to return the proper 404 response. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> update Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> precommit passing Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> remove header hotfix Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> precommit working Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more test coverage Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> add more test coverage Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net> test: adds real provider embeddings test & update doc (envoyproxy#841) **Description** This adds embeddings endpoint tests with the providers that support the endpoint. This only added the providers for which we have credentials. According to the testing situation we have right now, this also clarifies in the "Supported Endpoints" page that which provider is tested and which is not for each endpoint. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> cli: adds default route test (envoyproxy#842) **Description** This adds an additional test to aigw run command so that we can verify that setting the default route is possible. **Related Issues/PRs (if applicable)** Closes envoyproxy#612 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> build(deps): bump helm.sh/helm/v3 from 3.17.3 to 3.18.4 (envoyproxy#845) test: fixes TestStartConfigWatcher flake (envoyproxy#843) controller: ensure eg rollout when deployed as daemonset (envoyproxy#831) **Description** This PR handles the rollout for envoy gateway during ai gateway extproc upgrade when deployed as daemonset. Related Issues/PRs (if applicable) Related PR: envoyproxy#699 --------- Signed-off-by: Dan Sun <dsun20@bloomberg.net> make test var Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net>

**Description** This commit removes the handwritten header matching code from the extproc, and instead starts utilizing the hardened envoy native router. Historically, we had only one giant extproc filter where we did all logics including model name extraction, routing and then body transformation & upstream authorization. Since envoyproxy#599, we split into two external processor filters; one sits at the normal HTTP router and the other is configured at the per-cluster upstream HTTP filter. In theory, the one at HTTP router has only one job on request path: extracting model name from the request body. However, due to the historical reason, the handwritten router logic component remained, and that comes with not only a maintenance cost (forcing a complex extproc & control plane orchestration) but also a potential security vulnerability. In fact, writing header matching logic can be an easy attack surface, so if it's possible, we should avoid writing our own header matching (routing logic) but should rely on the battle-tested hardened envoy native router. With this commit, now a regex matching is available as well as there's no difference between HTTPRoute's matching and AIGatewayRoute's matching implementation. This also opens up a possibility to support path matching in our rule. **Related Issues/PRs (if applicable)** Ref envoyproxy#612 Ref envoyproxy#73 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net>

**Description** Before envoyproxy#793, the case where no matching route found was handled in the extproc and the 404 immediate response was returned from there, but after that, it naturally results in the "unreachable" default route and swallowed the indication of no matching and it made it impossible to reason about the 500 error on that case. In other words, this fixes the regression in envoyproxy#793 to return the proper 404 response. --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Signed-off-by: Alexa Griffith <agriffith50@bloomberg.net>

mathetake added this to the v0.3.0 milestone Jun 30, 2025

mathetake added 2 commits July 2, 2025 15:45

refactor: use Envoy native router

2e85202

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> # Conflicts: # api/v1alpha1/api.go # internal/controller/ai_gateway_route.go # internal/extproc/embeddings_processor.go

more

6421755

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake force-pushed the nativehttproute branch from 569c656 to 6421755 Compare July 2, 2025 22:46

mathetake added 6 commits July 2, 2025 16:09

more tests

62b7494

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

more

ba1b90c

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

fixes extproc test

ee2046b

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

fix translate

9e80201

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

more

7f4f1a8

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

more

69c796e

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake marked this pull request as ready for review July 3, 2025 00:41

mathetake requested a review from a team as a code owner July 3, 2025 00:41

unnecessary changes

78a7099

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

Xunzhuo approved these changes Jul 4, 2025

View reviewed changes

mathetake added 3 commits July 3, 2025 20:23

Merge remote-tracking branch 'origin/main' into nativehttproute

7105b3b

# Conflicts: # tests/crdcel/main_test.go

fix

ee80279

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

Merge remote-tracking branch 'origin/main' into nativehttproute

36be1dc

more coverage

e3c137a

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake merged commit 06d09b8 into main Jul 4, 2025
23 of 24 checks passed

mathetake deleted the nativehttproute branch July 4, 2025 04:30

mathetake mentioned this pull request Jul 7, 2025

controller: return 404 instead of 500 for no matching #837

Merged

mathetake mentioned this pull request Jul 8, 2025

default AIGatewayRoute backend #612

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: use Envoy native router#793

refactor: use Envoy native router#793
mathetake merged 13 commits intomainfrom
nativehttproute

mathetake commented Jun 28, 2025 •

edited

Loading

Uh oh!

Xunzhuo left a comment

Uh oh!

Xunzhuo commented Jul 4, 2025

Uh oh!

Xunzhuo commented Jul 4, 2025

Uh oh!

mathetake commented Jul 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mathetake commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Xunzhuo left a comment

Choose a reason for hiding this comment

Uh oh!

Xunzhuo commented Jul 4, 2025

Uh oh!

Xunzhuo commented Jul 4, 2025

Uh oh!

mathetake commented Jul 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mathetake commented Jun 28, 2025 •

edited

Loading