Skip to content

[model-gateway] introduce request ctx for oai router#14434

Merged
slin1237 merged 1 commit intomainfrom
oai-router-n/10
Dec 4, 2025
Merged

[model-gateway] introduce request ctx for oai router#14434
slin1237 merged 1 commit intomainfrom
oai-router-n/10

Conversation

@slin1237
Copy link
Copy Markdown
Collaborator

@slin1237 slin1237 commented Dec 4, 2025

introduce request context along with shared and responses component, such that we dont have to copy and paste component in every function signature

Checklist

@gemini-code-assist
Copy link
Copy Markdown
Contributor

Summary of Changes

Hello @slin1237, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a comprehensive refactoring of the OpenAI router by implementing a RequestContext object. This new structure centralizes the management and passing of common components and request-specific data, such as HTTP clients, storage interfaces, and MCP managers. The primary goal is to reduce boilerplate in function signatures, enhance code organization, and improve overall maintainability by providing a consistent and structured way to access dependencies throughout the request processing lifecycle.

Highlights

  • Introduction of RequestContext: A new context.rs file has been added, defining a RequestContext struct that centralizes request-specific input, shared components, and processing state for the OpenAI router pipeline. This includes RequestInput, RequestType, SharedComponents, ResponsesComponents, ComponentRefs, ProcessingState, WorkerSelection, PayloadState, StorageHandles, OwnedStreamingContext, and StreamingEventContext.
  • Refactoring of OpenAIRouter: The OpenAIRouter struct has been refactored to replace individual fields for HTTP client, storage interfaces, and MCP manager with SharedComponents and ResponsesComponents. This change streamlines dependency management within the router.
  • Simplified Function Signatures: Router methods such as handle_non_streaming_response, handle_streaming_response, and various conversation/response storage methods now accept or utilize the new RequestContext object. This significantly reduces the number of parameters passed to functions, improving readability and maintainability.
  • Provider Registry Enhancements: The ProviderRegistry now includes get_arc and default_provider_arc methods, which return Arc<dyn Provider>. This aligns with the new RequestContext structure for shared ownership of provider instances.
  • Streaming Module Integration: The streaming.rs module has been updated to leverage RequestContext and StreamingEventContext for passing context during streaming operations, further simplifying function signatures and centralizing data access in streaming handlers.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a RequestContext to refactor the OpenAI router, which is a great improvement for code clarity and maintainability. The implementation is solid, but I have a few suggestions. My main concerns are the removal of a significant number of tests, which should be restored, and the use of panics in accessor methods which could be replaced with Option/Result for better error handling. I've also noted some removed documentation that would be beneficial to restore.

I am having trouble creating individual review comments. Click here to see my feedback.

sgl-router/src/routers/openai/provider.rs (240-356)

high

A significant number of tests have been removed in this change. These tests covered important provider-specific logic, such as request transformations for different providers (OpenAI, XAI, Gemini) and provider registry lookups. Removing tests without replacing them with equivalent or better coverage is a regression and can lead to undetected bugs in the future.

Please restore these tests or move them to a more appropriate location if they have been relocated. If they are no longer relevant due to other changes, please provide a justification for their removal.

sgl-router/src/routers/openai/context.rs (60-92)

medium

The accessor methods mcp_manager, response_storage, conversation_storage, and conversation_item_storage panic if called on the Shared variant. This can lead to unrecoverable runtime errors if the component type is not what's expected.

To improve safety, consider changing these methods to return an Option or Result. This would make potential failures explicit at the call site, allowing for more robust error handling. The caller would then use .expect() or ? to handle the None case, making the panic point explicit or allowing the error to be propagated.

    pub fn mcp_manager(&self) -> Option<&Arc<McpManager>> {
        match self {
            ComponentRefs::Shared(_) => None,
            ComponentRefs::Responses(r) => Some(&r.mcp_manager),
        }
    }

    pub fn response_storage(&self) -> Option<&Arc<dyn ResponseStorage>> {
        match self {
            ComponentRefs::Shared(_) => None,
            ComponentRefs::Responses(r) => Some(&r.response_storage),
        }
    }

    pub fn conversation_storage(&self) -> Option<&Arc<dyn ConversationStorage>> {
        match self {
            ComponentRefs::Shared(_) => None,
            ComponentRefs::Responses(r) => Some(&r.conversation_storage),
        }
    }

    pub fn conversation_item_storage(&self) -> Option<&Arc<dyn ConversationItemStorage>> {
        match self {
            ComponentRefs::Shared(_) => None,
            ComponentRefs::Responses(r) => Some(&r.conversation_item_storage),
        }
    }

sgl-router/src/routers/openai/context.rs (181-206)

medium

The accessor methods worker, provider, payload, and take_payload use expect(), which will cause a panic if the required state is not set. This enforces a state machine-like flow but can make the code brittle if the state invariants are not perfectly maintained.

Similar to the feedback on ComponentRefs, consider returning an Option from these methods. This would move the responsibility of handling the missing state to the caller, making the code more robust and panics more explicit. The caller would then use .expect("..."), which clearly indicates a programmer error if the state is incorrect.

    pub fn worker(&self) -> Option<&Arc<dyn Worker>> {
        self.state.worker.as_ref().map(|w| &w.worker)
    }

    #[allow(dead_code)]
    pub fn provider(&self) -> Option<&dyn Provider> {
        self.state.worker.as_ref().map(|w| w.provider.as_ref())
    }

    pub fn payload(&self) -> Option<&PayloadState> {
        self.state.payload.as_ref()
    }

    pub fn take_payload(&mut self) -> Option<PayloadState> {
        self.state.payload.take()
    }

sgl-router/src/routers/openai/router.rs (247-258)

medium

This change removes the detailed documentation comment for the select_worker_for_model function. The removed comment provided valuable information about the function's logic, priorities, and behavior, which is important for maintainability. Other useful inline comments explaining the logic have also been removed.

Please restore the documentation comment for select_worker_for_model and other removed explanatory comments to ensure the code remains easy to understand and maintain for other developers.

@slin1237 slin1237 merged commit b01fc16 into main Dec 4, 2025
56 checks passed
@slin1237 slin1237 deleted the oai-router-n/10 branch December 4, 2025 16:31
tonyluj pushed a commit to openanolis/sglang that referenced this pull request Dec 5, 2025
tonyluj pushed a commit to openanolis/sglang that referenced this pull request Dec 5, 2025
yuchengz816-bot pushed a commit to yuchengz816-bot/sglang that referenced this pull request Dec 8, 2025
Kevin-XiongC pushed a commit to novitalabs/sglang that referenced this pull request Dec 9, 2025
dcampora pushed a commit to dcampora/sglang that referenced this pull request Dec 15, 2025
YChange01 pushed a commit to YChange01/sglang that referenced this pull request Jan 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant