Skip to content

Model config is inflexible #127

@lee-b

Description

@lee-b

This seems to only allow setting a global OPENAI_API_BASE and OPENAI_API_KEY in .env, and then VOICE_TOOLS_OPENAI_KEY. If so, it's not flexible enough: OpenAI themselves run a single API, but locally, it's likely that users would be running one inference server for their textgen model, another for TTS, another for ASR, each on different ports. So, there should at least be a default OPENAI_API_BASE/KEY and overrides per usage type: TTS_OPENAI_API_BASE/KEY, for example. Better approaches would be:

  1. Just add all (protocol, base_url, model) triplets. Then allow specifying the model by provider and names. This is akin to how litellm manages protocols, providers/endpoints/base_urls, and models.

  2. Add all protocols / providers / models per Terminal tool #1. Tag them according to capabilities. Weight automatically for preference, according to some heuristic (such as local first, more capable first, or more lightweight but sufficiently capable). Allow user override of tags and weightings. Auto select the best at runtime according to weight.

  3. Per two, but do everything up to heuristic weighting, then allow the user to override per usage type (TTS, ASR, Text gen, per agent, etc.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low — cosmetic, nice to havearea/configConfig system, migrations, profilestype/featureNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions