Overview

Add support for AWS Bedrock as a new provider in the Hub. This will allow users to route their LLM requests to AWS Bedrock models like Claude and Titan through our unified API interface.

Background

Currently, Hub supports OpenAI, Anthropic, Azure OpenAI and other providers (see src/providers/mod.rs). Adding AWS Bedrock support will expand the available model options and give users more flexibility in choosing their LLM providers, especially those already using AWS infrastructure.

Technical Details

New Provider Implementation

Create a new provider implementation in src/providers/bedrock/ following the existing provider pattern. Reference implementation can be found here.

Required Components

1. Create a new module structure (support streaming, non-streaming, tools and function calling, multi-modality)

   • src/providers/bedrock/mod.rs
   • src/providers/bedrock/provider.rs
   • src/providers/bedrock/models.rs

2. Implement the Provider trait with the following methods:

   • chat_completions
   • completions
   • embeddings

3. Add necessary request/response models for Bedrock API in models.rs

4. Update the provider registry to include Bedrock.

Configuration

5. Update the configuration structure to support bedrock, like:

providers:
  key: bedrock
type: bedrock
region: "<your-aws-region>"
access_key_id: "<your-aws-access-key-id>" # Optional if using IAM roles
secret_access_key: "<your-aws-secret-access-key>" # Optional if using IAM roles
models:
   key: claude-v2
type: anthropic.claude-v2
provider: bedrock
  key: titan
type: amazon.titan-text
provider: bedrock

6. Update the configuration example to include Bedrock settings.

Requirements

• Implement AWS authentication using either:
  • IAM role-based authentication
  • Access key/secret based authentication
• Support all major Bedrock models:
  • Anthropic Claude models
  • Amazon Titan models
  • AI21 Jurassic models
  • Stability.ai models
• Handle proper error mapping from AWS API responses to our status codes
• Implement proper request/response mapping for all endpoints
• Add comprehensive tests for the new provider
• Update documentation to include Bedrock setup instructions

Resources

• AWS Bedrock API Documentation
• AWS SDK Rust Documentation
• AWS Authentication Guide

Definition of Done

• Implementation of Bedrock provider
• Unit tests with good coverage
• Integration tests with actual API calls (using test credentials)
• Documentation updates
• Example configuration
• Successful chat completion, completion, and embedding requests
• Error handling and proper status code mapping
• PR review and approval

Additional Notes

• Feel free to ask questions in the issue comments
• We recommend creating a draft PR early to get feedback during implementation
• Make sure to follow the existing code style and patterns
• Don't commit any actual AWS credentials