This PR is stacked on top of all the prior work @chrsmith has done for shuffling configuration data around; it implements the new "Self hosted models" functionality.

Configuration

Configuring a Sourcegraph instance to use self-hosted models basically involves adding some configuration like this to the site config (if you set modelConfiguration, you are opting in to the new system which is in early access):

  // Setting this field means we are opting into the new Cody model configuration system.
  "modelConfiguration": {
    // Disable use of Sourcegraph's servers for model discovery
    "sourcegraph": null,

    // Create two model providers
    "providerOverrides": [
      {
        // Our first model provider "mistral" will be a Huggingface TGI deployment which hosts our
        // mistral model for chat functionality.
        "id": "mistral",
        "displayName": "Mistral",
        "serverSideConfig": {
          "type": "huggingface-tgi",
          "endpoints": [{"url": "https://mistral.example.com/v1"}]
        },
      },
      {
        // Our second model provider "bigcode" will be a Huggingface TGI deployment which hosts our
        // bigcode/starcoder model for code completion functionality.
        "id": "bigcode",
        "displayName": "Bigcode",
        "serverSideConfig": {
          "type": "huggingface-tgi",
          "endpoints": [{"url": "http://starcoder.example.com/v1"}]
        }
      }
    ],

    // Make these two models available to Cody users
    "modelOverridesRecommendedSettings": [
      "mistral::v1::mixtral-8x7b-instruct",
      "bigcode::v1::starcoder2-7b"
    ],

    // Configure which models Cody will use by default
    "defaultModels": {
      "chat": "mistral::v1::mixtral-8x7b-instruct",
      "fastChat": "mistral::v1::mixtral-8x7b-instruct",
      "codeCompletion": "bigcode::v1::starcoder2-7b"
    }
  }

More advanced configurations are possible, the above is our blessed configuration for today.

Hosting models

Another major component of this work is starting to build up recommendations around how to self-host models, which ones to use, how to configure them, etc.

For now, we've been testing with these two on a machine with dual A100s:

• Huggingface TGI (this is a Docker container for model inference, which provides an OpenAI-compatible API - and is widely popular)
• Two models:
  • Starcoder2 for code completion; specifically bigcode/starcoder2-15b with eetq 8-bit quantization.
  • Mixtral 8x7b instruct for chat; specifically casperhansen/mixtral-instruct-awq which uses awq 4-bit quantization.

This is our 'starter' configuration. Other models - specifically other starcoder 2, and mixtral instruct models - certainly work too, and higher parameter versions may of course provide better results.

Documentation for how to deploy Huggingface TGI, suggested configuration and debugging tips - coming soon.

Advanced configuration

As part of this effort, I have added a quite extensive set of configuration knobs to to the client side model configuration (see type ClientSideModelConfigOpenAICompatible in this PR)

Some of these configuration options are needed for things to work at a basic level, while others (e.g. prompt customization) are not needed for basic functionality, but are very important for customers interested in self-hosting their own models.

Today, Cody clients have a number of different autocomplete provider implementations which tie model-specific logic to enable autocomplete, to a provider. For example, if you use a GPT model through Azure OpenAI, the autocomplete provider for that is entirely different from what you'd get if you used a GPT model through OpenAI officially. This can lead to some subtle issues for us, and so it is worth exploring ways to have a generalized autocomplete provider - and since with self-hosted models we must address this problem, these configuration knobs fed to the client from the server are a pathway to doing that - initially just for self-hosted models, but in the future possibly generalized to other providers.

Debugging facilities

Working with customers in the past to use OpenAI-compatible APIs, we've learned that debugging can be quite a pain. If you can't see what requests the Sourcegraph backend is making, and what it is getting back.. it can be quite painful to debug.

This PR implements quite extensive logging, and a debugConnections flag which can be turned on to enable logging of the actual request payloads and responses. This is critical when a customer is trying to add support for a new model, their own custom OpenAI API service, etc.

Robustness

Working with customers in the past, we also learned that various parts of our backend openai provider were not super robust. For example, if more than one message was present it was a fatal error, or if the SSE stream yielded {"error"} payloads, they would go ignored. Similarly, the SSE event stream parser we use is heavily tailored towards the exact response structure which OpenAI's official API returns, and is therefor quite brittle if connecting to a different SSE stream.

For this work, I have started by forking our internal/completions/client/openai - and made a number of major improvements to it to make it more robust, handle errors better, etc.

I have also replaced the usage of a custom SSE event stream parser - which was not spec compliant and brittle - with a proper SSE event stream parser that recently popped up in the Go community: https://github.com/tmaxmax/go-sse

My intention is that after more extensive testing, this new internal/completions/client/openaicompatible provider will be more robust, more correct, and all around better than internal/completions/client/openai (and possibly the azure one) so that we can just supersede those with this new openaicompatible one entirely.

Client implementation

Much of the work done in this PR is just "let the site admin configure things, and broadcast that config to the client through the new model config system."

Actually getting the clients to respect the new configuration, is a task I am tackling in future sourcegraph/cody PRs.

Test plan

1. This change currently lacks any unit/regression tests, that is a major noteworthy point. I will follow-up with those in a future PR.

• However, these changes are incredibly isolated, clearly only affecting customers who opt-in to this new self-hosted models configuration.
• Most of the heavy lifting (SSE streaming, shuffling data around) is done in other well-tested codebases.

2. Manual testing has played a big role here, specifically:

• Running a dev instance with the new configuration, actually connected to Huggingface TGI deployed on a remote server.
• Using the new debugConnections mechanism (which customers would use) to directly confirm requests are going to the right places, with the right data and payloads.
• Confirming with a new client (changes not yet landed) that autocomplete and chat functionality work.

Can we use more testing? Hell yeah, and I'm going to add it soon. Does it work quite well and have small room for error? Also yes.

Changelog

Cody Enterprise: added a new configuration for self-hosting models. Reach out to support if you would like to use this feature as it is in early access.