perf: Prefer cached route hints during validation#156
Conversation
📝 WalkthroughWalkthroughRemoved context-based snapshot decoding logic from model validation and deleted the associated helper function. Updated tests to verify that caching does not occur when route hints are already present, shifting validation from caching canonical requests to relying on route hints for model/provider selection. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Poem
🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
internal/server/model_validation_test.go (1)
701-744: 🧹 Nitpick | 🔵 TrivialConsider asserting
RouteHints.Providerfor consistency.The first test (
DoesNotCacheCanonicalChatRequestWhenRouteHintsAlreadyExist) asserts bothRouteHints.ModelandRouteHints.Provider, but this test only assertsRouteHints.Model. PerstoreRequestModelResolution(context snippet 2), both fields are populated after resolution.Since the body doesn't include a
providerfield, you could either:
- Add
"provider":"openai"to the body and assert it matches, or- Assert the expected default/resolved provider value
This would make the test more complete and consistent with the chat request test pattern.
Option 1: Add provider to body and assert
frame := core.NewRequestSnapshot( http.MethodPost, "/v1/responses", nil, nil, nil, "application/json", []byte(`{ "model":"gpt-4o-mini", + "provider":"openai", "input":[{"type":"message","role":"user","content":"hi","x_trace":{"id":"trace-1"}}] }`), false, "", nil, )Then add assertion:
assert.Nil(t, capturedEnv.CachedResponsesRequest()) assert.Equal(t, "gpt-4o-mini", capturedEnv.RouteHints.Model) + assert.Equal(t, "openai", capturedEnv.RouteHints.Provider)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@internal/server/model_validation_test.go` around lines 701 - 744, Test TestModelValidation_DoesNotCacheCanonicalResponsesRequestWhenRouteHintsAlreadyExist is missing an assertion for RouteHints.Provider; update the test (in the function TestModelValidation_DoesNotCacheCanonicalResponsesRequestWhenRouteHintsAlreadyExist) to also assert the resolved provider on capturedEnv.RouteHints.Provider: either add "provider":"openai" to the JSON body so the handler/ExecutionPlanning resolution (storeRequestModelResolution / core.DeriveWhiteBoxPrompt) sets it explicitly and assert that value, or keep the body as-is and assert the expected default resolved provider (e.g., "openai") on capturedEnv.RouteHints.Provider to match the other test that checks both Model and Provider.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@internal/server/model_validation_test.go`:
- Around line 701-744: Test
TestModelValidation_DoesNotCacheCanonicalResponsesRequestWhenRouteHintsAlreadyExist
is missing an assertion for RouteHints.Provider; update the test (in the
function
TestModelValidation_DoesNotCacheCanonicalResponsesRequestWhenRouteHintsAlreadyExist)
to also assert the resolved provider on capturedEnv.RouteHints.Provider: either
add "provider":"openai" to the JSON body so the handler/ExecutionPlanning
resolution (storeRequestModelResolution / core.DeriveWhiteBoxPrompt) sets it
explicitly and assert that value, or keep the body as-is and assert the expected
default resolved provider (e.g., "openai") on capturedEnv.RouteHints.Provider to
match the other test that checks both Model and Provider.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: b6943796-041d-4df3-a78f-e86d86f77b9c
📒 Files selected for processing (2)
internal/server/model_validation.gointernal/server/model_validation_test.go
💤 Files with no reviewable changes (1)
- internal/server/model_validation.go
Summary
Describe what changed and why.
PR Title Format (required)
Use Conventional Commits in the PR title:
type(scope): short summaryExamples:
feat(cache): add Redis-backed model cachefix(streaming): flush done marker in SSEdocs(config): clarify provider auto-discoveryAllowed
typevalues:featfixperfdocsrefactortestbuildcichorerevertBreaking changes:
!before:(example:feat(api)!: remove legacy endpoint)Release Notes
feat,fix,perf, ordocs.test,ci,build,chore, manyrefactors) is auto-labeled and excluded from release notes.release:skiplabel to explicitly exclude an item from release notes.Summary by CodeRabbit