Skip to content

feat: implements CEL expression API for costs#153

Merged
mathetake merged 21 commits intomainfrom
celimpl
Jan 25, 2025
Merged

feat: implements CEL expression API for costs#153
mathetake merged 21 commits intomainfrom
celimpl

Conversation

@mathetake
Copy link
Copy Markdown
Member

@mathetake mathetake commented Jan 21, 2025

This commit implements the CEL expression API
left as a TODO in #103. Closes #97.

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
@mathetake mathetake marked this pull request as ready for review January 21, 2025 17:29
@mathetake mathetake requested a review from a team as a code owner January 21, 2025 17:29
@mathetake
Copy link
Copy Markdown
Member Author

@yuzisun PTAL

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Comment on lines -28 to +30
ModelNameHeaderKey, selectedBackendHeaderKey string
modelNameHeaderKey, selectedBackendHeaderKey string
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: shouldn't have been exported

p.costs.TotalTokens,
)
if err != nil {
return nil, fmt.Errorf("failed to evaluate CEL expression: %w", err)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, what does it mean for ext_proc to fail at this stage, when the response headers may have been sent? Does it make sense to make it fail?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will result in connection termination i guess

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
@mathetake
Copy link
Copy Markdown
Member Author

@envoyproxy/ai-gateway-maintainers ping

Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com>
@mathetake
Copy link
Copy Markdown
Member Author

@envoyproxy/ai-gateway-maintainers ping

Comment on lines +315 to +321
expr := *cost.CELExpression
// Sanity check the CEL expression.
_, err := llmcostcel.NewProgram(expr)
if err != nil {
return fmt.Errorf("invalid CEL expression: %w", err)
}
fc.CELExpression = expr
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we validate this in a webhook instead of controller as user is not going to get the error feedback if they specify the invalid cel expression ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

even if that's possible, still we need to do the type validation.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

even if that's possible, still we need to do the type validation.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me see how much k8s API can validate a CEL expression (not sure even if it's doable)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is doable, need to setup a validating web hook to achieve this, we can do this post 0.1 though.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, i did a quick research but is it really possible to write a k8s validation to verify a string field content itself being a valid CEL expression?

Copy link
Copy Markdown
Contributor

@yuzisun yuzisun Jan 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Essentially you add the same code in a webhook validation handler and then user can get the error feedback, though we still need this code to deal with the standalone setup with configmap.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok i see not the CRD CEL validation, but instead k8s validation webhook. let's do that later after v0.1.0

}

// Sanity check by evaluating the expression with some dummy values.
_, err = EvaluateProgram(prog, "dummy", "dummy", 0, 0, 0)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like getting called on the data path as well for each request, can we only initialize once?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prog is initialized on the config load time which is the compiled CEL program. so i don't think we could do beyond that (the other args are depending on the request context)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw to be clear, this function itself NewProgram is only called on config path and prog is initialized once here. And then on the request path only EvaluateProgram is called using the precompiled prog

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right I read the code wrong.

@netlify
Copy link
Copy Markdown

netlify bot commented Jan 25, 2025

Deploy Preview for envoy-ai-gateway ready!

Name Link
🔨 Latest commit 5cc068f
🔍 Latest deploy log https://app.netlify.com/sites/envoy-ai-gateway/deploys/679437de41d75e00080f11fb
😎 Deploy Preview https://deploy-preview-153--envoy-ai-gateway.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@mathetake mathetake merged commit 3f32432 into main Jan 25, 2025
@mathetake mathetake deleted the celimpl branch January 25, 2025 01:04
@mathetake
Copy link
Copy Markdown
Member Author

Thanks for the review Dan!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow customizing token cost calculation

3 participants