[model-gateway] add retry and circuit breaker support to gRPC routers#15585
[model-gateway] add retry and circuit breaker support to gRPC routers#15585
Conversation
- Add circuit breaker outcome recording to WorkerSelection in context.rs - Record outcomes in request_execution.rs after gRPC calls complete - Add RetryExecutor with exponential backoff to GrpcRouter and GrpcPDRouter - Use is_retryable_status for 408, 429, 500, 502, 503, 504 status codes - Record retry metrics with proper worker labels (WORKER_REGULAR, WORKER_PREFILL, WORKER_DECODE) - Consistent with HTTP router retry patterns
Summary of ChangesHello @slin1237, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly improves the robustness and reliability of the model-gateway's gRPC communication by integrating comprehensive retry and circuit breaker support. It introduces an exponential backoff retry strategy for gRPC requests across both standard and prefill/decode routers, coupled with outcome recording for worker selections to inform circuit breaking. These changes ensure that transient network issues or service unavailability are handled gracefully, reducing user-facing errors and providing clearer insights into system health through new metrics. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request effectively introduces retry and circuit breaker mechanisms to the gRPC routers, enhancing the resilience of the model gateway. The changes are well-structured, with clear separation of concerns: circuit breaker outcomes are recorded in the execution stage, and retry logic is encapsulated within RetryExecutor in the router implementations. The metric recording for retries and backoffs is also a great addition for observability.
My main feedback is regarding code duplication in both GrpcPDRouter and GrpcRouter where the retry logic is applied. I've left specific suggestions on how to refactor this into helper methods to improve maintainability. Overall, this is a solid contribution to making the gateway more robust.
Checklist