feat(dgw): add multi-provider AI Gateway #1588
Conversation
Adds an experimental AI Gateway feature that provides a unified proxy for multiple AI provider APIs through Devolutions Gateway. This enables centralized API key management, consistent authentication, and simplified access to various AI services from a single endpoint. The gateway supports seven providers with OpenAI-compatible and provider-specific endpoints, featuring transparent proxying, SSE streaming support, and flexible configuration through both JSON config and environment variables. Supported providers: - Mistral AI (cloud API) - OpenAI (cloud API) - Anthropic Claude (cloud API with custom message format) - OpenRouter (unified multi-provider API) - Azure OpenAI (deployment-based Azure service) - Ollama (local, OpenAI-compatible, optional auth) - LM Studio (local, OpenAI-compatible, optional auth) Key features: - Builder pattern for provider configuration with flexible auth methods - Generic proxy function supporting Bearer tokens and custom headers - Server-Sent Events (SSE) streaming for real-time completions - Gateway-level API key authentication (optional) - Environment variable overrides for all provider endpoints and keys - Configurable request timeout (default: 300s) - Transparent request/response proxying Architecture: - Provider-agnostic design with ProviderConfigBuilder - Support for Bearer auth (OpenAI, Mistral, OpenRouter) - Support for custom header auth (Anthropic x-api-key, Azure api-key) - Automatic injection of provider-specific headers - Full SSE streaming support across all providers Configuration follows existing patterns with JSON schema support and environment variable overrides (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.). Local providers (Ollama, LM Studio) default to localhost endpoints and have optional authentication. Requires enable_unstable: true in debug configuration. Issue: DGW-XXX
Let maintainers know that an action is required on their side
|
c7f1585 to
a8b6d1c
Compare
| /// Default HTTP client with a longer timeout suitable for AI requests. | ||
| fn create_client(timeout: Duration) -> reqwest::Client { | ||
| reqwest::Client::builder() | ||
| .timeout(timeout) | ||
| .build() | ||
| .expect("parameters known to be valid only") | ||
| } |
There was a problem hiding this comment.
note: I think it’s better to reuse the client instead of creating a new one for each request. We could use some sort of caching.
|
@copilot Implement some smoke tests in the testsuite crate. |
Adds an experimental AI Gateway feature that provides a unified proxy for multiple AI provider APIs through Devolutions Gateway. This enables centralized API key management, consistent authentication, and simplified access to various AI services from a single endpoint.
The gateway supports seven providers with OpenAI-compatible and provider-specific endpoints, featuring transparent proxying, SSE streaming support, and flexible configuration through both JSON config and environment variables.
Supported providers:
Key features:
Architecture:
Configuration follows existing patterns with JSON schema support and environment variable overrides (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.). Local providers (Ollama, LM Studio) default to localhost endpoints and have optional authentication.
Requires enable_unstable: true in debug configuration.