Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

Cody Gateway: Add support for Google non-streaming endpoint#63166

Merged
abeatrix merged 5 commits into
mainfrom
bee/gemini-chat
Jun 11, 2024
Merged

Cody Gateway: Add support for Google non-streaming endpoint#63166
abeatrix merged 5 commits into
mainfrom
bee/gemini-chat

Conversation

@abeatrix

@abeatrix abeatrix commented Jun 7, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Add support for non-stream request for Google Gemini provider

  • Added Stream field to googleRequest struct to enable streaming completions
  • Added SymtemInstruction field to googleRequest struct to allow setting system instructions
  • Updated GoogleHandlerMethods.validateRequest to allow FeatureEmbeddings instead of FeatureCodeCompletions
  • Updated GoogleHandlerMethods.getRequestMetadata to return the Stream field
  • Updated GoogleGatewayFeatureClient.GetRequest to handle streaming for both FeatureCodeCompletions and FeatureChatCompletions
  • Removed unsupported feature checks in googleCompletionStreamClient
  • Added Gemini 1.5 Flash and Gemini 1.0 Pro to autocomplete allowed list (but not supported by clients atm)

Test plan

Unit tests updated for non-stream request.

To manually test this:

  1. In your Soucegraph local instance's Site Config, add the following:
  "completions": {
    "accessToken": "REDACTED",
    "chatModel": "gemini-1.5-pro-latest",
    "completionModel": "google/gemini-1.5-flash-latest",
    "provider": "google",

Note: You can get the accessToken for Gemini API in 1Password.

  1. After saving the site config with the above change, run the following curl command that hits the code endpoint:
curl 'https://sourcegraph.test:3443/.api/completions/stream' -i \
-X POST \
-H 'authorization: token $YOUR_LOCAL_TOKEN' \
--data-raw '{"messages":[{"speaker":"human","text":"Who are you?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":false,"model":"gemini-1.5-pro-latest"}'

Output:

❯ curl 'https://sourcegraph.test:3443/.api/completions/stream' -i \
-X POST \
-H 'authorization: token $YOUR_LOCAL_TOKEN' \
--data-raw '{"messages":[{"speaker":"human","text":"Who are you?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":false,"model":"gemini-1.5-pro-latest"}'
HTTP/2 200
access-control-allow-credentials: true
access-control-allow-origin:
alt-svc: h3=":3443"; ma=2592000
cache-control: no-cache, max-age=0
content-type: text/plain; charset=utf-8
date: Tue, 11 Jun 2024 17:02:19 GMT
server: Caddy
server: Caddy
vary: Accept-Encoding, Authorization, Cookie, Authorization, X-Requested-With, Cookie
x-content-type-options: nosniff
x-frame-options: DENY
x-powered-by: Express
x-trace: e11a2ce292639414dd2ccdfcbfa89611
x-trace-span: 9457aa0dd0e09b6c
x-trace-url: https://sourcegraph.test:3443/-/debug/jaeger/trace/e11a2ce292639414dd2ccdfcbfa89611
x-xss-protection: 1; mode=block
content-length: 154

{"completion":"I am a large language model, trained by Google. \n\nHere's what that means:\n\n* **I am a computer program:** I","stopReason":"MAX_TOKENS"}%

Changelog

@abeatrix
Cody Gateway: add support for streaming completions and system instru…
5abc8e0 
…ctions to Gemini API

- Added `Stream` field to `googleRequest` struct to enable streaming completions
- Added `SymtemInstruction` field to `googleRequest` struct to allow setting system instructions
- Updated `GoogleHandlerMethods.validateRequest` to allow `FeatureEmbeddings` instead of `FeatureCodeCompletions`
- Updated `GoogleHandlerMethods.getRequestMetadata` to return the `Stream` field
- Updated `GoogleGatewayFeatureClient.GetRequest` to handle streaming for both `FeatureCodeCompletions` and `FeatureChatCompletions`
- Removed unsupported feature checks in `googleCompletionStreamClient`
@cla-bot cla-bot Bot added the cla-signed label Jun 7, 2024
@abeatrix abeatrix changed the title Cody Gateway: add support for code completions to Gemini API Cody Gateway: Add support for non-stream request for Google Jun 11, 2024
@abeatrix abeatrix marked this pull request as ready for review June 11, 2024 16:59
@abeatrix abeatrix requested review from a team, arafatkatze and emidoots June 11, 2024 17:22
emidoots
emidoots approved these changes Jun 11, 2024
View reviewed changes

@emidoots emidoots left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks great, thanks for adding tests - nice work!

@abeatrix abeatrix changed the title Cody Gateway: Add support for non-stream request for Google Cody Gateway: Add support for Google non-streaming endpoint Jun 11, 2024
@abeatrix abeatrix merged commit e155165 into main Jun 11, 2024
@abeatrix abeatrix deleted the bee/gemini-chat branch June 11, 2024 17:54
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@arafatkatze arafatkatze Awaiting requested review from arafatkatze

1 more reviewer

@emidoots emidoots emidoots approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla-signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@abeatrix @emidoots