cli: scaffolds 'aigw run' command by mathetake · Pull Request #498 · envoyproxy/ai-gateway

mathetake · 2025-03-14T19:50:11Z

Commit Message

This scaffolds aigw run [path] command that allows users to run AI Gateway natively without k8s/docker.

Related Issues/PRs (if applicable)

Extracted from #458
Contributes to #412

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake · 2025-03-14T21:32:02Z

envoyproxy/gateway#5505 is blocker for this

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake · 2025-03-18T15:55:52Z

another one: envoyproxy/gateway#5538

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake · 2025-03-18T19:26:24Z

🤞

mathetake · 2025-03-18T19:45:33Z

finally!!!!!!!!!!!!!!!!!!

mathetake · 2025-03-18T19:49:04Z

@aabchoo could you take a quick look? This is just a skeleton (not actually running/translationg the AI resources) and just adds the Envoy Gateway integration with an empty configuration extracted from #458. It is a bit large but the more than half is go.sum/go.mod and auto generated contents which you can ignore.

mathetake · 2025-03-18T19:49:50Z

manifests/charts/ai-gateway-helm/crds/aigateway.envoyproxy.io_backendsecuritypolicies.yaml

                                        maximum: 4294967295
                                        minimum: 0
                                        type: integer
+                                      perEndpoint:


this is due to the upgrades of EG to the latest, so not directly relevant

mathetake · 2025-03-18T19:50:21Z

go.mod

 	github.com/envoyproxy/go-control-plane/envoy v1.32.5-0.20250303232259-0c5cb90447bb
 	github.com/go-logr/logr v1.4.2
-	github.com/google/cel-go v0.24.1
+	github.com/google/cel-go v0.22.1


this was needed due to https://github.com/kubernetes/apiserver/issues/116 and need to pin it to 0.22.x until it's resolved

mathetake · 2025-03-18T19:50:36Z

cmd/aigw/run_test.go

+	"github.com/stretchr/testify/require"
+)
+
+func TestRun_default(t *testing.T) {


this will be expanded in the follow up #458

aabchoo · 2025-03-18T23:25:13Z

Makefile

 test-coverage:
 	@mkdir -p $(OUTPUT_DIR)
-	@$(MAKE) test GO_TEST_ARGS="-coverprofile=$(OUTPUT_DIR)/go-test-coverage.out -covermode=atomic -coverpkg=./... $(GO_TEST_ARGS)"
+	@$(MAKE) test GO_TEST_ARGS="-coverprofile=$(OUTPUT_DIR)/go-test-coverage.out -covermode=atomic -coverpkg=github.com/envoyproxy/ai-gateway/... $(GO_TEST_ARGS)"


curious: why this change?

yeah i should've left a note; envoyproxy/gateway is really a large project, hence generating the test coverage instrumented binary for it really take sa lot of time. So, simply this switches to do the instrumentation for the code in this repository (which is only thing we need in the first place for the test coverage report here).

cmd/aigw/run.go

aabchoo · 2025-03-18T23:38:13Z

cmd/aigw/main.go

+	// cmdRun corresponds to `aigw run` command.
+	cmdRun struct {
+		Debug bool   `help:"Enable debug logging emitted to stderr."`
+		Path  string `arg:"" name:"path" optional:"" help:"Path to the AI Gateway configuration yaml file. Optional. When this is not given, aigw runs the default configuration." type:"path"`


Do you think it would be helpful for users to know the default config file?

yeah that's my plan indeed. will do that in the next real implementation PR

aabchoo · 2025-03-18T23:40:47Z

cmd/aigw/certs/envoy/ca.crt

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDGDCCAgCgAwIBAgIEFZ8l4DANBgkqhkiG9w0BAQsFADAsMRYwFAYDVQQDEw1l
+bnZveS1nYXRld2F5MRIwEAYDVQQFEwkzNjI3NTE0NTYwHhcNMjUwMzA0MDAzODA4


Sanity check: do these certs never expire?

ok good point....

good catch and switched to do the cert gen on demand without checking them in. Now the code looks a bit simpler, thanks!

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake · 2025-03-19T01:00:26Z

cmd/aigw/run.go

+	// Even after the context is done, the goroutine managing the Envoy process might be still trying to shut it down.
+	// Give it some time to do so, otherwise the process might become an orphan. This is the limitation of the current
+	// API of func-e library that is used by Envoy Gateway to run the Envoy process.
+	// TODO: actually fix the library and the EG accordingly.


this will be done after envoyproxy/gateway#5527 lands

aabchoo

thanks for doing this!

cmd/aigw/run.go

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

**Commit Message** This implements the `aigw run` command that runs the AI Gateway locally. This utilizes the Envoy Gateway's standalone mode [1]. With this command, users are now able to use the AI Gateway, without Docker or Kubernetes. The follow up PRs to further modify the default configuration to add more de-facto services such as ollama, and will add the command to show the default configuration. 1: https://gateway.envoyproxy.io/docs/tasks/operations/standalone-deployment-mode/ **Related Issues/PRs (if applicable)** Contributes to #412 Follow up on #498 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake added 3 commits March 14, 2025 12:48

cli: scaffolds 'aigw run' command

c27686a

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

check in certs

61ab54e

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

more

a873c51

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake mentioned this pull request Mar 14, 2025

race condition between watcheabler.Map.Close and Subscribe envoyproxy/gateway#5505

Closed

mathetake added 4 commits March 14, 2025 13:41

see

400beca

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

see

b2ddca8

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

do not block

827bd35

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

recreate

8c05bc9

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake added 2 commits March 18, 2025 07:55

update

223ee17

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

update

7f54692

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

fix

18097ae

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake marked this pull request as ready for review March 18, 2025 19:46

mathetake requested a review from a team as a code owner March 18, 2025 19:46

mathetake commented Mar 18, 2025

View reviewed changes

mathetake assigned aabchoo Mar 18, 2025

aabchoo reviewed Mar 18, 2025

View reviewed changes

mathetake added 3 commits March 18, 2025 17:00

typo

4bdb83a

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

do the certgen on demand

21df2cd

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

more

2862b9f

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

mathetake commented Mar 19, 2025

View reviewed changes

aabchoo approved these changes Mar 19, 2025

View reviewed changes

cmd/aigw/run.go Outdated Show resolved Hide resolved

cmd/aigw/run.go Outdated Show resolved Hide resolved

mathetake added 2 commits March 19, 2025 08:16

more

0c377fa

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>

Merge remote-tracking branch 'origin/main' into initialruncommand

ab7a870

mathetake enabled auto-merge (squash) March 19, 2025 15:17

mathetake merged commit e5c298e into main Mar 19, 2025
14 of 15 checks passed

mathetake deleted the initialruncommand branch March 19, 2025 15:18

mathetake mentioned this pull request Mar 20, 2025

cli: initial implementation of aigw run #458

Merged

Conversation

mathetake commented Mar 14, 2025

Uh oh!

mathetake commented Mar 14, 2025

Uh oh!

mathetake commented Mar 18, 2025

Uh oh!

mathetake commented Mar 18, 2025

Uh oh!

mathetake commented Mar 18, 2025

Uh oh!

mathetake commented Mar 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mathetake Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aabchoo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mathetake Mar 18, 2025 •

edited

Loading