[model-gateway] introduce request ctx for oai router#14434
Conversation
Summary of ChangesHello @slin1237, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a comprehensive refactoring of the OpenAI router by implementing a Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a RequestContext to refactor the OpenAI router, which is a great improvement for code clarity and maintainability. The implementation is solid, but I have a few suggestions. My main concerns are the removal of a significant number of tests, which should be restored, and the use of panics in accessor methods which could be replaced with Option/Result for better error handling. I've also noted some removed documentation that would be beneficial to restore.
I am having trouble creating individual review comments. Click here to see my feedback.
sgl-router/src/routers/openai/provider.rs (240-356)
A significant number of tests have been removed in this change. These tests covered important provider-specific logic, such as request transformations for different providers (OpenAI, XAI, Gemini) and provider registry lookups. Removing tests without replacing them with equivalent or better coverage is a regression and can lead to undetected bugs in the future.
Please restore these tests or move them to a more appropriate location if they have been relocated. If they are no longer relevant due to other changes, please provide a justification for their removal.
sgl-router/src/routers/openai/context.rs (60-92)
The accessor methods mcp_manager, response_storage, conversation_storage, and conversation_item_storage panic if called on the Shared variant. This can lead to unrecoverable runtime errors if the component type is not what's expected.
To improve safety, consider changing these methods to return an Option or Result. This would make potential failures explicit at the call site, allowing for more robust error handling. The caller would then use .expect() or ? to handle the None case, making the panic point explicit or allowing the error to be propagated.
pub fn mcp_manager(&self) -> Option<&Arc<McpManager>> {
match self {
ComponentRefs::Shared(_) => None,
ComponentRefs::Responses(r) => Some(&r.mcp_manager),
}
}
pub fn response_storage(&self) -> Option<&Arc<dyn ResponseStorage>> {
match self {
ComponentRefs::Shared(_) => None,
ComponentRefs::Responses(r) => Some(&r.response_storage),
}
}
pub fn conversation_storage(&self) -> Option<&Arc<dyn ConversationStorage>> {
match self {
ComponentRefs::Shared(_) => None,
ComponentRefs::Responses(r) => Some(&r.conversation_storage),
}
}
pub fn conversation_item_storage(&self) -> Option<&Arc<dyn ConversationItemStorage>> {
match self {
ComponentRefs::Shared(_) => None,
ComponentRefs::Responses(r) => Some(&r.conversation_item_storage),
}
}sgl-router/src/routers/openai/context.rs (181-206)
The accessor methods worker, provider, payload, and take_payload use expect(), which will cause a panic if the required state is not set. This enforces a state machine-like flow but can make the code brittle if the state invariants are not perfectly maintained.
Similar to the feedback on ComponentRefs, consider returning an Option from these methods. This would move the responsibility of handling the missing state to the caller, making the code more robust and panics more explicit. The caller would then use .expect("..."), which clearly indicates a programmer error if the state is incorrect.
pub fn worker(&self) -> Option<&Arc<dyn Worker>> {
self.state.worker.as_ref().map(|w| &w.worker)
}
#[allow(dead_code)]
pub fn provider(&self) -> Option<&dyn Provider> {
self.state.worker.as_ref().map(|w| w.provider.as_ref())
}
pub fn payload(&self) -> Option<&PayloadState> {
self.state.payload.as_ref()
}
pub fn take_payload(&mut self) -> Option<PayloadState> {
self.state.payload.take()
}sgl-router/src/routers/openai/router.rs (247-258)
This change removes the detailed documentation comment for the select_worker_for_model function. The removed comment provided valuable information about the function's logic, priorities, and behavior, which is important for maintainability. Other useful inline comments explaining the logic have also been removed.
Please restore the documentation comment for select_worker_for_model and other removed explanatory comments to ensure the code remains easy to understand and maintain for other developers.
3b1b076 to
5fb0c5f
Compare
5fb0c5f to
d0e8a1f
Compare
Co-authored-by: key4ng <rukeyang@gmail.com>
Co-authored-by: key4ng <rukeyang@gmail.com>
Co-authored-by: key4ng <rukeyang@gmail.com>
Co-authored-by: key4ng <rukeyang@gmail.com>
Co-authored-by: key4ng <rukeyang@gmail.com>
Co-authored-by: key4ng <rukeyang@gmail.com>
introduce request context along with shared and responses component, such that we dont have to copy and paste component in every function signature
Checklist