Skip to content

perf: Prefer cached route hints during validation#156

Merged
SantiagoDePolonia merged 1 commit intomainfrom
perf/improvements
Mar 18, 2026
Merged

perf: Prefer cached route hints during validation#156
SantiagoDePolonia merged 1 commit intomainfrom
perf/improvements

Conversation

@SantiagoDePolonia
Copy link
Copy Markdown
Contributor

@SantiagoDePolonia SantiagoDePolonia commented Mar 18, 2026

Summary

Describe what changed and why.

PR Title Format (required)

Use Conventional Commits in the PR title:

type(scope): short summary

Examples:

  • feat(cache): add Redis-backed model cache
  • fix(streaming): flush done marker in SSE
  • docs(config): clarify provider auto-discovery

Allowed type values:

  • feat
  • fix
  • perf
  • docs
  • refactor
  • test
  • build
  • ci
  • chore
  • revert

Breaking changes:

  • Add ! before : (example: feat(api)!: remove legacy endpoint)

Release Notes

  • User-facing work should use feat, fix, perf, or docs.
  • Internal-only work (test, ci, build, chore, many refactors) is auto-labeled and excluded from release notes.
  • Use release:skip label to explicitly exclude an item from release notes.

Summary by CodeRabbit

  • Refactor
    • Streamlined model validation logic to optimize request processing when route hints are provided.

@github-actions github-actions bot added the release:other Other release-noteworthy changes label Mar 18, 2026
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 18, 2026

📝 Walkthrough

Walkthrough

Removed context-based snapshot decoding logic from model validation and deleted the associated helper function. Updated tests to verify that caching does not occur when route hints are already present, shifting validation from caching canonical requests to relying on route hints for model/provider selection.

Changes

Cohort / File(s) Summary
Context-based Decoding Removal
internal/server/model_validation.go
Deleted decodeCanonicalSelectorHintsForValidation helper function and removed the conditional path that invoked it. Function now falls back to parsing from request body JSON and existing environment hints. Context import removed.
Test Assertions Update
internal/server/model_validation_test.go
Renamed two tests to reflect non-caching behavior when route hints exist. Updated assertions to verify that cached requests are nil and that RouteHints contain expected Model and Provider instead of checking cached canonical content.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐰 A snapshot once danced in the context so deep,
But logic evolved, decisions to keep,
Now hints guide the way, no cache to delay,
Route wisdom prevails—a cleaner display! ✨

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description is incomplete—it only contains the template structure without actual content describing the change, impact, or rationale. Replace the template placeholder with a meaningful summary explaining what changed, why caching was optimized, and the performance impact of preferring cached route hints.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: removing context-based snapshot decoding and preferring cached route hints during validation instead.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch perf/improvements
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@SantiagoDePolonia SantiagoDePolonia changed the title pefor: Prefer cached route hints during validation perf: Prefer cached route hints during validation Mar 18, 2026
@github-actions github-actions bot added release:performance Performance improvement for release notes and removed release:other Other release-noteworthy changes labels Mar 18, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
internal/server/model_validation_test.go (1)

701-744: 🧹 Nitpick | 🔵 Trivial

Consider asserting RouteHints.Provider for consistency.

The first test (DoesNotCacheCanonicalChatRequestWhenRouteHintsAlreadyExist) asserts both RouteHints.Model and RouteHints.Provider, but this test only asserts RouteHints.Model. Per storeRequestModelResolution (context snippet 2), both fields are populated after resolution.

Since the body doesn't include a provider field, you could either:

  1. Add "provider":"openai" to the body and assert it matches, or
  2. Assert the expected default/resolved provider value

This would make the test more complete and consistent with the chat request test pattern.

Option 1: Add provider to body and assert
 	frame := core.NewRequestSnapshot(
 		http.MethodPost,
 		"/v1/responses",
 		nil,
 		nil,
 		nil,
 		"application/json",
 		[]byte(`{
 			"model":"gpt-4o-mini",
+			"provider":"openai",
 			"input":[{"type":"message","role":"user","content":"hi","x_trace":{"id":"trace-1"}}]
 		}`),
 		false,
 		"",
 		nil,
 	)

Then add assertion:

 	assert.Nil(t, capturedEnv.CachedResponsesRequest())
 	assert.Equal(t, "gpt-4o-mini", capturedEnv.RouteHints.Model)
+	assert.Equal(t, "openai", capturedEnv.RouteHints.Provider)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@internal/server/model_validation_test.go` around lines 701 - 744, Test
TestModelValidation_DoesNotCacheCanonicalResponsesRequestWhenRouteHintsAlreadyExist
is missing an assertion for RouteHints.Provider; update the test (in the
function
TestModelValidation_DoesNotCacheCanonicalResponsesRequestWhenRouteHintsAlreadyExist)
to also assert the resolved provider on capturedEnv.RouteHints.Provider: either
add "provider":"openai" to the JSON body so the handler/ExecutionPlanning
resolution (storeRequestModelResolution / core.DeriveWhiteBoxPrompt) sets it
explicitly and assert that value, or keep the body as-is and assert the expected
default resolved provider (e.g., "openai") on capturedEnv.RouteHints.Provider to
match the other test that checks both Model and Provider.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@internal/server/model_validation_test.go`:
- Around line 701-744: Test
TestModelValidation_DoesNotCacheCanonicalResponsesRequestWhenRouteHintsAlreadyExist
is missing an assertion for RouteHints.Provider; update the test (in the
function
TestModelValidation_DoesNotCacheCanonicalResponsesRequestWhenRouteHintsAlreadyExist)
to also assert the resolved provider on capturedEnv.RouteHints.Provider: either
add "provider":"openai" to the JSON body so the handler/ExecutionPlanning
resolution (storeRequestModelResolution / core.DeriveWhiteBoxPrompt) sets it
explicitly and assert that value, or keep the body as-is and assert the expected
default resolved provider (e.g., "openai") on capturedEnv.RouteHints.Provider to
match the other test that checks both Model and Provider.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: b6943796-041d-4df3-a78f-e86d86f77b9c

📥 Commits

Reviewing files that changed from the base of the PR and between 06ed504 and 2d82113.

📒 Files selected for processing (2)
  • internal/server/model_validation.go
  • internal/server/model_validation_test.go
💤 Files with no reviewable changes (1)
  • internal/server/model_validation.go

@SantiagoDePolonia SantiagoDePolonia merged commit 6cfce3d into main Mar 18, 2026
18 of 19 checks passed
@SantiagoDePolonia SantiagoDePolonia deleted the perf/improvements branch April 4, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release:performance Performance improvement for release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant