Overview

Add support for Google VertexAI models as a new provider in the Hub. This will allow users to route their LLM requests to Google's models like Gemini through our unified API interface.

Background

Currently, Hub supports OpenAI, Anthropic, and Azure OpenAI providers (see src/providers/mod.rs). Adding Google VertexAI support will expand the available model options and give users more flexibility in choosing their LLM providers.

Technical Details

New Provider Implementation

Create a new provider implementation in src/providers/vertexai/ following the existing provider pattern. Reference implementation can be found here

Required Components

1. Create a new module structure:

   • src/providers/vertexai/mod.rs
   • src/providers/vertexai/provider.rs
   • src/providers/vertexai/models.rs

2. Implement the Provider trait with the following methods (support streaming, non-streaming, tools, functions, multi modality etc...)

   • chat_completions
   • completions
   • embeddings

3. Add necessary request/response models for VertexAI API in models.rs

4. Update the provider registry to include VertexAI

Configuration

6. Add a new provider section:

providers:
  key: vertexai
type: vertexai
project_id: "<your-gcp-project-id>"
location: "<your-gcp-region>"
credentials_path: "/path/to/service-account.json" # Optional

7. Update the configuration example to include VertexAI settings.

Requirements

• Implement authentication using Google Cloud credentials
• Support both API key and service account authentication methods
• Handle proper error mapping from Google API responses to our status codes
• Implement proper request/response mapping for all endpoints
• Add comprehensive tests for the new provider
• Update documentation to include VertexAI setup instructions

Resources

• Google VertexAI API Documentation
• Gemini API Reference
• Google Cloud Authentication Guide

Definition of Done

• Implementation of VertexAI provider
• Unit tests with good coverage
• Integration tests with actual API calls (using test credentials)
• Documentation updates
• Example configuration
• Successful chat completion, completion, and embedding requests
• Error handling and proper status code mapping
• PR review and approval

Additional Notes

• Feel free to ask questions in the issue comments
• We recommend creating a draft PR early to get feedback during implementation
• Make sure to follow the existing code style and patterns
• Don't commit any actual API keys or credentials