wip(cody): Introduce cody-gateway-config tool for generating model configuration#63324
Conversation
| The output of this tool is a `.json` file which we check into this repo. It will | ||
| be embedded in Go binaries as a simple way of deploying the configuration blob. | ||
|
|
||
| ## Usage |
There was a problem hiding this comment.
Yes, this is kinda clunky and can probably be replaced by a sufficiently awesome bezel command or sg plugin. If there's a good example for how to do something like that, I'd be happy to update it in this PR.
But otherwise I'd move that, and adding a CI/CD check to verify that there is "no diff" if rerunning the tool, to a follow-up PR.
There was a problem hiding this comment.
can probably be replaced by a sufficiently awesome bezel command or sg plugin.
Fly-by comment: IMO no need for sg plugin, just putting it directly in sg is the way to go, per the sg mission of being "the one tool you need to install and run"
| anthropic_06_2023 = "anthropic::2023-06-01" | ||
| ) | ||
|
|
||
| return []types.Model{ |
There was a problem hiding this comment.
This is unfortunately clunky, and the initial version of this tool (written in TypeScript) was much more elegant.
However, I rewrote it in Go because:
While it would be nice to have the TypeScript and Golang projections of the data types exist side-by-side, having a dependency (in either direction) between the sourcegraph/sourcegraph and sourcegraph/cody repos would be kind of a pain.
Also, we would never be in a situation where we would update the TypeScript types (used by Cody clients to read the configuration data fetched from the backend) before the Golang types (which will persist/read/write the configuration data on the backend).
Similarly, there would be no need for the actual logic (such as validating the document) in the TypeScript side because for Cody clients it would only need to read the data. (And could trust that it was well-formed when sent by the server.)
So it seemed like the net best approach would be to put the Golang-types in the monorepo, along with the cody-gateway-config tool to simplify "deployment" of its output. And then put the TypeScript types directly in the sourcegraph/cody repo. (Similar to how it defines types for things it receives from the Sourcegraph GraphQL API, etc.)
| ModelRef ModelRef `json:"modelRef"` | ||
|
|
||
| DisplayName string `json:"displayName"` | ||
| ModelName string `json:"string"` |
There was a problem hiding this comment.
All of these fields should have a big doc comment explaining what they mean and are used for. (e.g. the distinction between Model.ID and Model.ModelName.)
But it didn't seem like we'd want to have the super-verbose names in both the Golang and TypeScript types. And they would be much more useful on the Cody Client-side. (Since that's where we would actually use/care about the values.)
So I could easily be convinced to add all of the missing comments in this PR. But that was generally the thinking to keeping it spartan. (Except for the configuration data, which will only ever be used within the Sourcegraph instance. So definitely we need to have a descriptive comment on top of AWSBedrockProviderConfig .
Since it's not specific to gateway (also served embedded in the server binary); it might make sense to remove gateway from the name. related.. I would suggest just calling this |
| "modelRef": "anthropic::2023-06-01::claude-3-sonnet", | ||
| "displayName": "Claude 3 Sonnet", | ||
| "string": "claude-3-sonnet-20240229", | ||
| "capabilities": ["chat", "edit"], |
There was a problem hiding this comment.
It'd be really nice if we can move away from 'edit' in favor of 'codeCompletion' or similar
This PR adds a couple of foundational components of the server-side LLM configuration project. (Server-side Cody Model Selection)
cody-gateway-config, which will write a JSON file to disk.internal/modelconfigpackage for defining LLM Model Configuration data.What this enables
The
cody-gateway-configcontains logic for building aModelConfigurationobject in-memory, and saving it to disk. The resulting JSON file is intended to be persisted in the repository with the namecody-gateway-model-configuration.json.The purpose of this tool is to track the current list of LLM models that Cody Gateway supports in-code, so it can be versioned as part of the regular code-review and PR process. (Generally, this is just a fancier way to generate the same data that is in the
sourcegraph/codyrepo'sdotcom.tsfile.)With the file being checked into the repo, we can then statically embed it into any Golang binaries that reference the
internal/modelconfig/embeddedpackage. This is to make it super-easy to distribute the latest Cody Gateway configuration data, since we do not need to copy the file and/or figure out how to load it from the local disk. And instead, just have the read-only contents available in-memory. (If size ever becomes a concern, we could embed a compressed version, etc.)Why that is needed
After this PR gets merged, both Cody Gateway and the Sourcegraph instance codebases will be able to take a dependency on this
modelconfig/embeddedpackage and use this data as part of the larger "server-side LLM model configuration" story.There's a lot that I'm glossing over in that list. But the big picture is that we will use these data types, and static
.jsonfile to power how this information will flow between various Sourcegraph components.Test plan
Added unit tests for validating the generated configuration data is valid.
Changelog
NA, this is an initial step. A later PR will elaborate on the exciting new feature this is enabling.