Cody Gateway: Add Gemini models to PLG and Enterprise users by abeatrix · Pull Request #63053 · sourcegraph/sourcegraph-public-snapshot

abeatrix · 2024-06-03T22:06:26Z

CLOSE https://github.com/sourcegraph/cody-issues/issues/211 & https://github.com/sourcegraph/cody-issues/issues/412 & https://github.com/sourcegraph/cody-issues/issues/412
UNBLOCK https://github.com/sourcegraph/cody/pull/4360

Add support for Google Gemini AI models as chat completions provider
Add new google package to handle Google Generative AI client
Update client.go and codygateway.go to handle the new Google provider
Set default models for chat, fast chat, and completions when Google is the configured provider
Add gemini-pro to the allowed list

Test plan

For Enterprise instances using google as provider:

In your Soucegraph local instance's Site Config, add the following:

    "accessToken": "REDACTED",
    "chatModel": "gemini-1.5-pro-latest",
    "provider": "google",

Note: You can get the accessToken for Gemini API in 1Password.

After saving the site config with the above change, run the following curl command:

curl 'https://sourcegraph.test:3443/.api/completions/stream' -i \
-X POST \
-H 'authorization: token $LOCAL_INSTANCE_TOKEN' \
--data-raw '{"messages":[{"speaker":"human","text":"Who are you?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":true,"model":"gemini-1.5-pro-latest"}'

Expected Output:

❯ curl 'https://sourcegraph.test:3443/.api/completions/stream' -i \
-X POST \
-H 'authorization: token <REDACTED>' \
--data-raw '{"messages":[{"speaker":"human","text":"Who are you?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":true,"model":"gemini-1.5-pro-latest"}'

HTTP/2 200
access-control-allow-credentials: true
access-control-allow-origin:
alt-svc: h3=":3443"; ma=2592000
cache-control: no-cache
content-type: text/event-stream
date: Tue, 04 Jun 2024 05:45:33 GMT
server: Caddy
server: Caddy
vary: Accept-Encoding, Authorization, Cookie, Authorization, X-Requested-With, Cookie
x-accel-buffering: no
x-content-type-options: nosniff
x-frame-options: DENY
x-powered-by: Express
x-trace: d4b1f02a3e2882a3d52331335d217b03
x-trace-span: 728ec33860d3b5e6
x-trace-url: https://sourcegraph.test:3443/-/debug/jaeger/trace/d4b1f02a3e2882a3d52331335d217b03
x-xss-protection: 1; mode=block

event: completion
data: {"completion":"I","stopReason":"STOP"}

event: completion
data: {"completion":"I am a large language model, trained by Google. \n\nThink of me as","stopReason":"STOP"}

event: completion
data: {"completion":"I am a large language model, trained by Google. \n\nThink of me as a computer program that can understand and generate human-like text.","stopReason":"MAX_TOKENS"}

event: done
data: {}

Verified locally:

Before

Cody Gateway returns no client known for upstream provider google

curl -X 'POST' -d '{"messages":[{"speaker":"human","text":"Who are you?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":true,"model":"google/gemini-1.5-pro-latest"}' -H 'Accept: application/json' -H 'Authorization: token $YOUR_DOTCOM_TOKEN' -H 'Content-Type: application/json' 'https://sourcegraph.com/.api/completions/stream'

event: error
data: {"error":"no client known for upstream provider google"}

event: done
data: {

Changelog

Added support for Google as an LLM provider for Cody, with the following models available through Cody Gateway: Gemini Pro (gemini-pro-latest), Gemini 1.5 Flash (gemini-1.5-flash-latest), and Gemini 1.5 Pro (gemini-1.5-pro-latest).

* Add support for Google Generative AI as a completions provider * Add new `google` package to handle Google Generative AI client * Update `client.go` and `codygateway.go` to handle the new Google provider * Set default models for chat, fast chat, and completions when Google is the configured provider

vdavid · 2024-06-04T21:03:55Z

 - A feature flag for Cody, `completions.smartContextWindow` is added and set to "enabled" by default. It allows clients to adjust the context window based on the name of the chat model. When smartContextWindow is enabled, the `completions.chatModelMaxTokens` value is ignored. ([#62802](https://github.com/sourcegraph/sourcegraph/pull/62802))
 - Code Insights: When facing the "incomplete datapoints" warning, you can now use GraphQL to discover which repositories had problems. The schemas for `TimeoutDatapointAlert` and `GenericIncompleteDatapointAlert` now contain an additional `repositories` field. ([#62756](https://github.com/sourcegraph/sourcegraph/pull/62756)).
 - Users will now be presented with a modal that reminds them to connect any external code host accounts that's required for permissions. Without these accounts connected, users may be unable to view repositories that they otherwise have access to. [#62983](https://github.com/sourcegraph/sourcegraph/pull/62983)
+- Added support for Google as an LLM provider for Cody, with the following models available through Cody Gateway: Gemini Pro (`gemini-pro-latest`), Gemini 1.5 Flash (`gemini-1.5-flash-latest`), and Gemini 1.5 Pro (`gemini-1.5-pro-latest`). [#63053](https://github.com/sourcegraph/sourcegraph/pull/63053)


Is this something we still do manually? Doesn't the "# Changelog" section of the PR take care of it?

@vdavid didn't know that, TIL!

It was a genuine question. 😄 I saw the template in the PR, so I've started using it, but never checked whether some automated or manual process actually added my changes to the changelog. Right now I'm just blindly relying on the assumption that some process will take care of it.

vdavid

I went through all files in detail, and TBH, it just LGTM. Saw the test coverage where there was significant logic, and that you ran this code and worked for you, so I don't have anything to add, it looks good. Great job, Bee! :)

chrsmith

Some nit picks and suggestions for cleanup, but this all looks good to me!

abeatrix · 2024-06-04T23:46:21Z

sg start from the latest commit and verified the changes still work correctly:

HTTP/2 200
access-control-allow-credentials: true
access-control-allow-origin:
alt-svc: h3=":3443"; ma=2592000
cache-control: no-cache
content-type: text/event-stream
date: Tue, 04 Jun 2024 23:44:44 GMT
server: Caddy
server: Caddy
vary: Accept-Encoding, Authorization, Cookie, Authorization, X-Requested-With, Cookie
x-accel-buffering: no
x-content-type-options: nosniff
x-frame-options: DENY
x-powered-by: Express
x-trace: 73f3b2564bd04916e4f10c237b93fd3d
x-trace-span: e846063737030d46
x-trace-url: https://sourcegraph.test:3443/-/debug/jaeger/trace/73f3b2564bd04916e4f10c237b93fd3d
x-xss-protection: 1; mode=block

event: completion
data: {"completion":"I","stopReason":"STOP"}

event: completion
data: {"completion":"I am a large language model, trained by Google. \n\nHere are some of","stopReason":"STOP"}

event: completion
data: {"completion":"I am a large language model, trained by Google. \n\nHere are some of the things I can do:\n\n**Communication and Language:**\n\n","stopReason":"MAX_TOKENS"}

event: done
data: {}

cla-bot Bot added the cla-signed label Jun 3, 2024

abeatrix requested a review from chrsmith June 3, 2024 22:06

abeatrix added 3 commits June 3, 2024 15:48

simplify the URL construction

1d5b8be

update bazel files

aa85429

remove feature flag cody-pro-gemini-enabled

275e73b

abeatrix changed the title ~~CODY PLG: add support for Google Generative AI~~ Cody Gateway: Add support for Gemini to PLG and Enterprise Jun 4, 2024

Add google to provider list

f5136d2

abeatrix requested review from emidoots and rafax June 4, 2024 06:33

Merge branch 'main' into bee/gemini-plg

4ce5fa3

abeatrix changed the title ~~Cody Gateway: Add support for Gemini to PLG and Enterprise~~ Cody Gateway: Add Gemini models to PLG and Enterprise users Jun 4, 2024

Add Gemini Pro

f4c3232

abeatrix force-pushed the bee/gemini-plg branch from 3e67ddc to f4c3232 Compare June 4, 2024 19:25

abeatrix marked this pull request as ready for review June 4, 2024 19:26

abeatrix requested review from a team June 4, 2024 19:27

Add CHANGELOG entry

ac5b77a

vdavid reviewed Jun 4, 2024

View reviewed changes

vdavid approved these changes Jun 4, 2024

View reviewed changes

chrsmith approved these changes Jun 4, 2024

View reviewed changes

Apply code review feedback

099cda1

abeatrix enabled auto-merge (squash) June 4, 2024 23:46

abeatrix merged commit f2590cb into main Jun 4, 2024

abeatrix deleted the bee/gemini-plg branch June 4, 2024 23:46

abeatrix mentioned this pull request Jun 6, 2024

Cody PLG: Add Gemini 1.5 Pro and Gemini 1.5 Flash models sourcegraph/cody-public-snapshot#4360

Merged

abeatrix mentioned this pull request Nov 25, 2024

Add modelConfig docs sourcegraph/docs#735

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cody Gateway: Add Gemini models to PLG and Enterprise users#63053

Cody Gateway: Add Gemini models to PLG and Enterprise users#63053
abeatrix merged 9 commits into
mainfrom
bee/gemini-plg

abeatrix commented Jun 3, 2024 •

edited

Loading

Uh oh!

vdavid Jun 4, 2024

Uh oh!

abeatrix Jun 4, 2024

Uh oh!

vdavid Jun 4, 2024

Uh oh!

vdavid left a comment

Uh oh!

chrsmith left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abeatrix commented Jun 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

abeatrix commented Jun 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test plan

Before

Changelog

Uh oh!

vdavid Jun 4, 2024

Choose a reason for hiding this comment

Uh oh!

abeatrix Jun 4, 2024

Choose a reason for hiding this comment

Uh oh!

vdavid Jun 4, 2024

Choose a reason for hiding this comment

Uh oh!

vdavid left a comment

Choose a reason for hiding this comment

Uh oh!

chrsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abeatrix commented Jun 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

abeatrix commented Jun 3, 2024 •

edited

Loading