feat: Add provider switch notification feature#23
Merged
Conversation
This feature adds user-facing notifications when LunaRoute switches
providers due to rate limits, server errors, or circuit breaker events.
Key design decisions:
- Prepend user message (compatible with OpenAI and Claude)
- Trigger on: rate limits (429), 5xx errors, circuit breaker open
- Generic, professional default message
- Global config with per-provider overrides
- Template variables: ${new_provider}, ${original_provider}, ${reason}, ${model}
- Enabled by default
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add SwitchReason enum to categorize provider switches: - RateLimit: 429 errors - ServiceIssue: 5xx errors - CircuitBreaker: circuit breaker open Each reason maps to a generic user-facing message: - RateLimit -> "high demand" - ServiceIssue -> "a temporary service issue" - CircuitBreaker -> "service maintenance" Includes comprehensive unit tests for enum functionality and cloning. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add configuration struct for provider switch notifications with: - enabled flag (defaults to true via serde) - default_message field with professional default template - Default trait implementation - Serde serialization/deserialization support Tests verify: - Default configuration contains IMPORTANT prefix - Serialization roundtrip works - enabled defaults to true when not specified 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add public re-exports for ProviderSwitchNotificationConfig and SwitchReason to make notification types accessible to consumers of lunaroute-routing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add provider_switch_notification field to RoutingConfig struct - Field is optional and defaults to None - Add tests for serialization with and without notification config - Tests verify YAML deserialization works correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add switch_notification_message field to OpenAIConfig - Add switch_notification_message field to AnthropicConfig - Update constructors to initialize field with None default - Add tests for both configs to verify field works - Update all test constructors to include new field - Update server provider initialization with None default - Update integration test constructors with None default 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Adds substitute_template_variables function that replaces template variables
in notification messages with actual values:
- ${original_provider}: Provider that failed
- ${new_provider}: Provider being switched to
- ${reason}: Generic reason for switch
- ${model}: Model name from request
Includes 5 comprehensive tests covering all variables, no variables, partial
variables, duplicates, and special characters.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds build_notification_message function that constructs notification messages by combining template selection and variable substitution: - Uses provider-specific override message if available - Falls back to global default message - Substitutes all template variables (provider names, reason, model) Includes 3 tests covering default message, provider override, and variable substitution. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Adds has_notification_already function that prevents duplicate notifications during cascading failovers by checking if the first message already starts with "IMPORTANT:". Includes 4 comprehensive tests: - Notification present (returns true) - Notification absent (returns false) - Empty messages array (returns false) - Multimodal message (returns false, doesn't check non-text content) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…cations Complete Phase 4 tasks from provider-switch-notification implementation plan: Task 4.1: Add notification config to Router struct - Added notification_config field to Router struct - Updated Router::new() to accept notification_config parameter - Updated Router::with_defaults() to pass None for notification_config - Updated all Router::new() test calls with None parameter Task 4.2: Implement provider config lookup - Added get_provider_notification_message() method to Router - Method retrieves custom notification message from provider configs - Initially stubbed, then completed in Task 4.3 Task 4.3: Extend Provider trait - Added get_notification_message() method to Provider trait with default impl - Implemented for OpenAIConnector returning switch_notification_message - Implemented for AnthropicConnector returning switch_notification_message - Updated Router's get_provider_notification_message() to use trait method Task 4.4: Implement inject notification logic - Added inject_notification_if_needed() method to Router - Checks if notifications are enabled via config - Uses has_notification_already() for idempotency guard - Builds notification using build_notification_message() - Prepends notification message to request messages array - Returns true if notification was injected, false otherwise All tests passing: - lunaroute-routing: 147 tests passed - lunaroute-core: 44 tests passed - lunaroute-egress: 17 tests passed - lunaroute_integration_tests: 27 tests passed Note: Methods are currently unused (marked with #[allow(dead_code)]) because Phase 5 integration into error handling flow is not yet implemented. This is intentional and per the implementation plan. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit integrates the notification injection system into the actual provider switching flow at three critical points: 1. **Rate limit switching (LimitsAlternative strategy)**: When a primary provider returns a rate limit error and the router switches to an alternative provider, a notification is injected into the request explaining the switch due to "high demand". 2. **Fallback switching (5xx errors, circuit breaker, other errors)**: When the primary provider fails and the router tries fallback providers, a notification is injected based on the error type: - 5xx errors trigger "service issue" notifications - Circuit breaker open triggers "service maintenance" notifications - Other errors default to "service issue" notifications 3. **Streaming fallback (circuit breaker)**: When streaming requests are routed to fallback providers due to circuit breaker state, a notification is injected with the "service maintenance" reason. Implementation details: - Added `is_5xx_error()` helper method to detect server errors by checking for 5xx status codes and error messages - Removed `#[allow(dead_code)]` attributes from notification methods - Notification injection occurs by cloning the request, prepending the notification message, and passing the modified request to the alternative or fallback provider - Error type flags (`is_rate_limit_error`, `is_5xx_error`) are declared before the match and assigned within the error arm to ensure they remain in scope for the fallback logic All existing tests continue to pass, demonstrating that the integration maintains backward compatibility with current routing behavior. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Issue 1: Simplified circuit breaker reason logic - Removed misleading circuit breaker check in fallback switch_reason - Circuit breaker check was always false after trying provider - Now uses ServiceIssue for all non-rate-limit errors - Removed unused is_5xx_error function as distinction no longer needed Issue 2: Eliminated need for 5xx detection - Both 5xx and non-5xx errors now use ServiceIssue switch reason - Simplified logic to only distinguish rate limits from other errors - Removed is_5xx_error variable and function entirely Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Test rate limit switch injects notification - Test notification can be disabled - Test cross-dialect notification (OpenAI → Anthropic) - Test provider-specific custom message overrides - Test idempotency guard prevents duplicate notifications All 5 tests passing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Demonstrates global notification configuration - Shows provider-specific custom messages with template variables - Includes GPT models with OpenAI→Anthropic rate limit protection - Includes Claude models with Anthropic→OpenAI fallback - Documents all notification feature capabilities 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Added section in Advanced Configuration - Documented features, configuration, and template variables - Included reference to example config file - Shows both global and per-provider customization options 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Changes: - Move metrics initialization before router creation in main.rs - Change Router::with_defaults() to Router::new() to accept notification config - Pass config.routing.provider_switch_notification to Router - Add log message when provider switch notifications are enabled This completes Phase 7 Task 1 - server now properly passes notification configuration from YAML/database config through to the routing layer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements automatic user notifications when LunaRoute switches providers due to rate limits, errors, or circuit breaker events. The notification is injected as a prepended user message that instructs the LLM to inform the end user about the provider switch.
Features
Implementation Details
Core Components
Data Types (
crates/lunaroute-routing/src/notification.rs)SwitchReasonenum: RateLimit, ServiceError, CircuitBreaker, MaintenanceProviderSwitchNotificationConfig: Global on/off and default message${original_provider},${new_provider},${reason},${model}Notification Logic
build_notification_message(): Constructs notification with variable substitutionhas_notification(): Idempotency guard using "IMPORTANT:" prefix detectionRouter Integration (
crates/lunaroute-routing/src/provider_router.rs)Server Configuration (
crates/lunaroute-server/src/config.rs,main.rs)provider_switch_notificationtoRoutingConfigswitch_notification_messageto provider configsConfiguration Example
Testing
Unit Tests (15 tests)
Integration Tests (5 tests)
Test Results
Documentation
examples/configs/provider-switch-notification.yamlFiles Changed
Core Implementation:
crates/lunaroute-routing/src/notification.rs(new, 234 lines)crates/lunaroute-routing/src/provider_router.rs(modified)crates/lunaroute-core/src/provider.rs(modified)crates/lunaroute-egress/src/openai/mod.rs(modified)crates/lunaroute-egress/src/anthropic/mod.rs(modified)crates/lunaroute-server/src/config.rs(modified)crates/lunaroute-server/src/main.rs(modified)Tests:
crates/lunaroute-integration-tests/tests/switch_notification_integration.rs(new, 671 lines)crates/lunaroute-integration-tests/tests/limits_alternative_strategy.rs(new, 739 lines)Documentation:
examples/configs/provider-switch-notification.yaml(new, 108 lines)README.md(modified, +38 lines)Technical Highlights
Breaking Changes
None. This feature is opt-in via configuration.
Migration Guide
To enable provider switch notifications:
Verification
✅ All tests passing
✅ Release build successful
✅ Server loads config correctly
✅ Log output confirms notifications enabled
✅ Manual testing completed
Ready for review and merge.
🤖 Generated with Claude Code
Co-Authored-By: Claude noreply@anthropic.com