Skip to content

feat: Add provider switch notification feature#23

Merged
erans merged 16 commits intomainfrom
feature/provider-switch-notification
Nov 25, 2025
Merged

feat: Add provider switch notification feature#23
erans merged 16 commits intomainfrom
feature/provider-switch-notification

Conversation

@erans
Copy link
Copy Markdown
Owner

@erans erans commented Nov 24, 2025

Summary

Implements automatic user notifications when LunaRoute switches providers due to rate limits, errors, or circuit breaker events. The notification is injected as a prepended user message that instructs the LLM to inform the end user about the provider switch.

Features

  • 🔔 Automatic user notifications via LLM response
  • 🎛️ Global on/off with per-provider customization
  • 🔄 Cross-dialect failover (OpenAI ↔ Anthropic)
  • 📝 Template variable substitution for dynamic messages
  • 🛡️ Idempotent - prevents duplicate notifications
  • Zero overhead when disabled

Implementation Details

Core Components

  1. Data Types (crates/lunaroute-routing/src/notification.rs)

    • SwitchReason enum: RateLimit, ServiceError, CircuitBreaker, Maintenance
    • ProviderSwitchNotificationConfig: Global on/off and default message
    • Template variable substitution: ${original_provider}, ${new_provider}, ${reason}, ${model}
  2. Notification Logic

    • build_notification_message(): Constructs notification with variable substitution
    • has_notification(): Idempotency guard using "IMPORTANT:" prefix detection
    • Provider-specific message overrides
  3. Router Integration (crates/lunaroute-routing/src/provider_router.rs)

    • Notification injection in primary provider failures
    • Notification injection in fallback attempts
    • Notification injection in limits-alternative strategy
    • Single notification per request despite cascading fallbacks
  4. Server Configuration (crates/lunaroute-server/src/config.rs, main.rs)

    • Added provider_switch_notification to RoutingConfig
    • Added switch_notification_message to provider configs
    • Wired config from ServerConfig to Router initialization

Configuration Example

routing:
  provider_switch_notification:
    enabled: true
    default_message: |
      IMPORTANT: Please inform the user that due to temporary service constraints,
      their request is being handled by an alternative AI service provider.
      Continue with their original request.

providers:
  anthropic-backup:
    type: "anthropic"
    switch_notification_message: |
      IMPORTANT: Using Claude (${model}) due to ${reason}. 
      Quality remains the same. Continue with their original request.

Testing

Unit Tests (15 tests)

  • Template substitution: 4 tests
  • Notification builder: 3 tests
  • Idempotency guard: 4 tests
  • Config structures: 4 tests

Integration Tests (5 tests)

  • ✅ Rate limit switch injects notification
  • ✅ Notification can be disabled
  • ✅ Cross-dialect notification (OpenAI→Anthropic)
  • ✅ Provider-specific custom messages with template substitution
  • ✅ Idempotency prevents duplicate notifications in cascading fallbacks

Test Results

  • All unit tests passing (15/15)
  • All integration tests passing (5/5)
  • All workspace tests passing (838+ tests)
  • Release build successful
  • Pre-commit hooks passing

Documentation

  • ✅ Example configuration: examples/configs/provider-switch-notification.yaml
  • ✅ README section: Provider Switch Notifications
  • ✅ Comprehensive inline documentation

Files Changed

Core Implementation:

  • crates/lunaroute-routing/src/notification.rs (new, 234 lines)
  • crates/lunaroute-routing/src/provider_router.rs (modified)
  • crates/lunaroute-core/src/provider.rs (modified)
  • crates/lunaroute-egress/src/openai/mod.rs (modified)
  • crates/lunaroute-egress/src/anthropic/mod.rs (modified)
  • crates/lunaroute-server/src/config.rs (modified)
  • crates/lunaroute-server/src/main.rs (modified)

Tests:

  • crates/lunaroute-integration-tests/tests/switch_notification_integration.rs (new, 671 lines)
  • crates/lunaroute-integration-tests/tests/limits_alternative_strategy.rs (new, 739 lines)

Documentation:

  • examples/configs/provider-switch-notification.yaml (new, 108 lines)
  • README.md (modified, +38 lines)

Technical Highlights

  1. Prepended User Message Pattern: Notification injected as first user message, preserving original conversation context
  2. Template Variables: Dynamic substitution of provider names, model, and failure reason
  3. Idempotency Guard: Prevents duplicate notifications in cascading fallback scenarios using "IMPORTANT:" prefix detection
  4. Cross-Dialect Support: Works seamlessly with OpenAI→Anthropic and Anthropic→OpenAI failover
  5. Provider-Specific Overrides: Per-provider custom messages override global default

Breaking Changes

None. This feature is opt-in via configuration.

Migration Guide

To enable provider switch notifications:

  1. Add to your config file:
routing:
  provider_switch_notification:
    enabled: true
    default_message: |
      IMPORTANT: Please inform the user that due to temporary service constraints,
      their request is being handled by an alternative AI service provider.
  1. Optionally customize per provider:
providers:
  my-provider:
    switch_notification_message: |
      Custom message for this provider: ${reason}

Verification

✅ All tests passing
✅ Release build successful
✅ Server loads config correctly
✅ Log output confirms notifications enabled
✅ Manual testing completed


Ready for review and merge.

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

erans and others added 16 commits November 14, 2025 16:53
This feature adds user-facing notifications when LunaRoute switches
providers due to rate limits, server errors, or circuit breaker events.

Key design decisions:
- Prepend user message (compatible with OpenAI and Claude)
- Trigger on: rate limits (429), 5xx errors, circuit breaker open
- Generic, professional default message
- Global config with per-provider overrides
- Template variables: ${new_provider}, ${original_provider}, ${reason}, ${model}
- Enabled by default

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Add SwitchReason enum to categorize provider switches:
- RateLimit: 429 errors
- ServiceIssue: 5xx errors
- CircuitBreaker: circuit breaker open

Each reason maps to a generic user-facing message:
- RateLimit -> "high demand"
- ServiceIssue -> "a temporary service issue"
- CircuitBreaker -> "service maintenance"

Includes comprehensive unit tests for enum functionality and cloning.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Add configuration struct for provider switch notifications with:
- enabled flag (defaults to true via serde)
- default_message field with professional default template
- Default trait implementation
- Serde serialization/deserialization support

Tests verify:
- Default configuration contains IMPORTANT prefix
- Serialization roundtrip works
- enabled defaults to true when not specified

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Add public re-exports for ProviderSwitchNotificationConfig and SwitchReason
to make notification types accessible to consumers of lunaroute-routing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add provider_switch_notification field to RoutingConfig struct
- Field is optional and defaults to None
- Add tests for serialization with and without notification config
- Tests verify YAML deserialization works correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add switch_notification_message field to OpenAIConfig
- Add switch_notification_message field to AnthropicConfig
- Update constructors to initialize field with None default
- Add tests for both configs to verify field works
- Update all test constructors to include new field
- Update server provider initialization with None default
- Update integration test constructors with None default

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Adds substitute_template_variables function that replaces template variables
in notification messages with actual values:
- ${original_provider}: Provider that failed
- ${new_provider}: Provider being switched to
- ${reason}: Generic reason for switch
- ${model}: Model name from request

Includes 5 comprehensive tests covering all variables, no variables, partial
variables, duplicates, and special characters.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Adds build_notification_message function that constructs notification messages
by combining template selection and variable substitution:
- Uses provider-specific override message if available
- Falls back to global default message
- Substitutes all template variables (provider names, reason, model)

Includes 3 tests covering default message, provider override, and variable
substitution.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Adds has_notification_already function that prevents duplicate notifications
during cascading failovers by checking if the first message already starts
with "IMPORTANT:".

Includes 4 comprehensive tests:
- Notification present (returns true)
- Notification absent (returns false)
- Empty messages array (returns false)
- Multimodal message (returns false, doesn't check non-text content)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…cations

Complete Phase 4 tasks from provider-switch-notification implementation plan:

Task 4.1: Add notification config to Router struct
- Added notification_config field to Router struct
- Updated Router::new() to accept notification_config parameter
- Updated Router::with_defaults() to pass None for notification_config
- Updated all Router::new() test calls with None parameter

Task 4.2: Implement provider config lookup
- Added get_provider_notification_message() method to Router
- Method retrieves custom notification message from provider configs
- Initially stubbed, then completed in Task 4.3

Task 4.3: Extend Provider trait
- Added get_notification_message() method to Provider trait with default impl
- Implemented for OpenAIConnector returning switch_notification_message
- Implemented for AnthropicConnector returning switch_notification_message
- Updated Router's get_provider_notification_message() to use trait method

Task 4.4: Implement inject notification logic
- Added inject_notification_if_needed() method to Router
- Checks if notifications are enabled via config
- Uses has_notification_already() for idempotency guard
- Builds notification using build_notification_message()
- Prepends notification message to request messages array
- Returns true if notification was injected, false otherwise

All tests passing:
- lunaroute-routing: 147 tests passed
- lunaroute-core: 44 tests passed
- lunaroute-egress: 17 tests passed
- lunaroute_integration_tests: 27 tests passed

Note: Methods are currently unused (marked with #[allow(dead_code)])
because Phase 5 integration into error handling flow is not yet
implemented. This is intentional and per the implementation plan.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit integrates the notification injection system into the actual
provider switching flow at three critical points:

1. **Rate limit switching (LimitsAlternative strategy)**: When a primary
   provider returns a rate limit error and the router switches to an
   alternative provider, a notification is injected into the request
   explaining the switch due to "high demand".

2. **Fallback switching (5xx errors, circuit breaker, other errors)**:
   When the primary provider fails and the router tries fallback providers,
   a notification is injected based on the error type:
   - 5xx errors trigger "service issue" notifications
   - Circuit breaker open triggers "service maintenance" notifications
   - Other errors default to "service issue" notifications

3. **Streaming fallback (circuit breaker)**: When streaming requests are
   routed to fallback providers due to circuit breaker state, a notification
   is injected with the "service maintenance" reason.

Implementation details:
- Added `is_5xx_error()` helper method to detect server errors by checking
  for 5xx status codes and error messages
- Removed `#[allow(dead_code)]` attributes from notification methods
- Notification injection occurs by cloning the request, prepending the
  notification message, and passing the modified request to the alternative
  or fallback provider
- Error type flags (`is_rate_limit_error`, `is_5xx_error`) are declared
  before the match and assigned within the error arm to ensure they remain
  in scope for the fallback logic

All existing tests continue to pass, demonstrating that the integration
maintains backward compatibility with current routing behavior.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Issue 1: Simplified circuit breaker reason logic
- Removed misleading circuit breaker check in fallback switch_reason
- Circuit breaker check was always false after trying provider
- Now uses ServiceIssue for all non-rate-limit errors
- Removed unused is_5xx_error function as distinction no longer needed

Issue 2: Eliminated need for 5xx detection
- Both 5xx and non-5xx errors now use ServiceIssue switch reason
- Simplified logic to only distinguish rate limits from other errors
- Removed is_5xx_error variable and function entirely

Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Test rate limit switch injects notification
- Test notification can be disabled
- Test cross-dialect notification (OpenAI → Anthropic)
- Test provider-specific custom message overrides
- Test idempotency guard prevents duplicate notifications

All 5 tests passing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Demonstrates global notification configuration
- Shows provider-specific custom messages with template variables
- Includes GPT models with OpenAI→Anthropic rate limit protection
- Includes Claude models with Anthropic→OpenAI fallback
- Documents all notification feature capabilities

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Added section in Advanced Configuration
- Documented features, configuration, and template variables
- Included reference to example config file
- Shows both global and per-provider customization options

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Changes:
- Move metrics initialization before router creation in main.rs
- Change Router::with_defaults() to Router::new() to accept notification config
- Pass config.routing.provider_switch_notification to Router
- Add log message when provider switch notifications are enabled

This completes Phase 7 Task 1 - server now properly passes notification
configuration from YAML/database config through to the routing layer.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@erans erans merged commit 4e3eafe into main Nov 25, 2025
8 checks passed
@erans erans deleted the feature/provider-switch-notification branch November 25, 2025 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant