Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

Adding Anthropic messages API support to the Google provider through Google vertex#63282

Merged
emidoots merged 13 commits into
mainfrom
google-vertex
Jun 20, 2024
Merged

Adding Anthropic messages API support to the Google provider through Google vertex#63282
emidoots merged 13 commits into
mainfrom
google-vertex

Conversation

@arafatkatze

@arafatkatze arafatkatze commented Jun 16, 2024

Copy link
Copy Markdown
Contributor

Linear Issue

This PR adds support for anthropic models in the google provider through google vertex.
NOTE: The current code only supported Google Gemini API and had boiler plate code for Google vertex(only for the gemini model) this PR adds Google Vertex for anthropic models properly so this way the google provider can be run in 3 different configurations

  1. Google Gemini API(this works but only for chat and not for completions which is the intended behaviour for now)
  2. Google Vertex API Anthropic Model(This works perfectly and is added in this PR and tested for both chat and completions and it works great)
  3. Google Vertex API Gemini Model (this doesn't work yet and can eventually be added to this codebase but we gotta add a new decoder for the streaming responses of the gemini model through this API we can take care of this later)

Sense of Urgency: This is a P0 because of enterprise requirements so I would appreciate a fast approval and merging.

Test plan

Changelog

@cla-bot cla-bot Bot added the cla-signed label Jun 16, 2024
@arafatkatze arafatkatze changed the title Google vertex WIP Work in progress Google vertex Jun 16, 2024
@arafatkatze arafatkatze force-pushed the google-vertex branch 5 times, most recently from 1fa215c to a714961 Compare June 17, 2024 16:20
@arafatkatze arafatkatze changed the title WIP Work in progress Google vertex Adding Anthropic messages API support to the Google provider through Google vertex Jun 19, 2024
Comment thread internal/completions/client/google/gemini_types.go Outdated
Comment thread internal/completions/client/google/google.go Outdated
Comment thread internal/completions/client/google/google.go Outdated
Comment thread internal/completions/client/google/gemini_types.go Outdated
Comment thread internal/completions/client/google/google.go Outdated
Comment thread internal/completions/client/google/google.go Outdated
Comment thread internal/completions/client/google/google.go Outdated
@emidoots

Copy link
Copy Markdown
Member

looks like you've got a bit of cleanup left to do here, and a few merge conflicts to resolve - can you push commits to fix all my comments before our call tomorrow?

@arafatkatze arafatkatze force-pushed the google-vertex branch 2 times, most recently from 05fbd36 to a4f6836 Compare June 20, 2024 12:36
abeatrix and others added 2 commits June 20, 2024 14:42
- Validate that the input messages are not empty
- Ensure the first message is a non-empty assistant message
- Skip empty assistant messages at the end of the prompt
- Disallow consistent speaker role between consecutive messages
- Add tests for the various validation cases
Comment thread internal/completions/client/google/anthropic_types.go Outdated
Comment thread internal/completions/client/google/anthropic_types.go Outdated
Comment thread internal/completions/client/google/google.go
Comment thread internal/completions/client/google/google.go Outdated
Comment thread internal/completions/client/google/google.go Outdated
Comment thread internal/completions/client/google/google.go Outdated
func getgRPCMethod(stream bool) string {
func getgRPCMethod(stream bool, modelFamily ModelFamily) string {
if modelFamily == VertexAnthropic {
return "streamRawPredict"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems sus, shouldn't this be under the if stream case below? otherwise streamRawPredict would be used, which is presumably for streaming only(?), even when the non-streaming request is being made

@emidoots

Copy link
Copy Markdown
Member

Left a few more thoughts - let's go over them and fix them together in a call if that helps

Stephen Gutekanst and others added 7 commits June 20, 2024 09:06
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
@emidoots emidoots merged commit 1a6a7f7 into main Jun 20, 2024
@emidoots emidoots deleted the google-vertex branch June 20, 2024 17:50
@DaedalusG DaedalusG added backports backport 5.4.5099 label used to backport PRs to the 5.4.5099 release branch labels Jun 20, 2024
@sourcegraph-release-bot

sourcegraph-release-bot commented Jun 20, 2024

Copy link
Copy Markdown
Collaborator

The backport to 5.4.5099 failed at https://github.com/sourcegraph/sourcegraph/actions/runs/9602082248:

The process '/usr/bin/git' failed with exit code 1

To backport this PR manually, you can either:

Via the sg tool

Use the sg backport command to backport your commit to the release branch.

sg backport -r 5.4.5099 -p 63282
Via your terminal

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-5.4.5099 5.4.5099
# Navigate to the new working tree
cd .worktrees/backport-5.4.5099
# Create a new branch
git switch --create backport-63282-to-5.4.5099
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1a6a7f78bf5b545f16e604e4879a56b14f3a12a4
# Push it to GitHub
git push --set-upstream origin backport-63282-to-5.4.5099
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-5.4.5099

If you encouter conflict, first resolve the conflict and stage all files, then run the commands below:

git cherry-pick --continue
# Push it to GitHub
git push --set-upstream origin backport-63282-to-5.4.5099
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-5.4.5099
  • Follow above instructions to backport the commit.
  • Create a pull request where the base branch is 5.4.5099 and the compare/head branch is backport-63282-to-5.4.5099., click here to create the pull request.

Once the pull request has been created, please ensure the following:

  • Make sure to tag @sourcegraph/release in the pull request description.

  • kindly remove the release-blocker from this pull request.

@sourcegraph-release-bot sourcegraph-release-bot added failed-backport-to-5.4.5099 release-blocker Prevents us from releasing: https://about.sourcegraph.com/handbook/engineering/releases labels Jun 20, 2024
Chickensoupwithrice pushed a commit that referenced this pull request Jun 20, 2024
…Google vertex (#63282)

[Linear
Issue](https://linear.app/sourcegraph/project/claude-3-on-gcp-8c014e1a3506/overview)

This PR adds support for anthropic models in the google provider through
google vertex.
NOTE: The current code only supported Google Gemini API and had boiler
plate code for Google vertex(only for the gemini model) this PR adds
Google Vertex for anthropic models properly so this way the google
provider can be run in 3 different configurations
1. Google Gemini API(this works but only for chat and not for
completions which is the intended behaviour for now)
2. Google Vertex API Anthropic Model(This works perfectly and is added
in this PR and tested for both chat and completions and it works great)
3. Google Vertex API Gemini Model (this doesn't work yet and can
eventually be added to this codebase but we gotta add a new decoder for
the streaming responses of the gemini model through this API we can take
care of this later)

Sense of Urgency: This is a P0 because of enterprise requirements so I
would appreciate a fast approval and merging.

<!-- 💡 To write a useful PR description, make sure that your description
covers:
- WHAT this PR is changing:
    - How was it PREVIOUSLY.
    - How it will be from NOW on.
- WHY this PR is needed.
- CONTEXT, i.e. to which initiative, project or RFC it belongs.

The structure of the description doesn't matter as much as covering
these points, so use
your best judgement based on your context.
Learn how to write good pull request description:
https://www.notion.so/sourcegraph/Write-a-good-pull-request-description-610a7fd3e613496eb76f450db5a49b6e?pvs=4
-->

- Run this branch for Cody instance ->
https://github.com/sourcegraph/cody/pull/4606
- Ask @arafatkatze to dm you the siteadmin config to make things work
- Check the logs and play with completions and chat

<!-- All pull requests REQUIRE a test plan:
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->

<!--
1. Ensure your pull request title is formatted as: $type($domain): $what
3. Add bullet list items for each additional detail you want to cover
(see example below)
4. You can edit this after the pull request was merged, as long as
release shipping it hasn't been promoted to the public.
5. For more information, please see this how-to
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c?

Audience: TS/CSE > Customers > Teammates (in that order).

Cheat sheet: $type = chore|fix|feat $domain:
source|search|ci|release|plg|cody|local|...
-->

<!--
Example:

Title: fix(search): parse quotes with the appropriate context
Changelog section:

- When a quote is used with regexp pattern type, then ...
- Refactored underlying code.
-->

---------

Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Co-authored-by: Beatrix <beatrix@sourcegraph.com>
Co-authored-by: Stephen Gutekanst <stephen@sourcegraph.com>
(cherry picked from commit 1a6a7f7)
@Chickensoupwithrice Chickensoupwithrice restored the google-vertex branch June 20, 2024 18:46
Chickensoupwithrice pushed a commit that referenced this pull request Jun 20, 2024
…Google vertex (#63282)

[Linear
Issue](https://linear.app/sourcegraph/project/claude-3-on-gcp-8c014e1a3506/overview)

This PR adds support for anthropic models in the google provider through
google vertex.
NOTE: The current code only supported Google Gemini API and had boiler
plate code for Google vertex(only for the gemini model) this PR adds
Google Vertex for anthropic models properly so this way the google
provider can be run in 3 different configurations
1. Google Gemini API(this works but only for chat and not for
completions which is the intended behaviour for now)
2. Google Vertex API Anthropic Model(This works perfectly and is added
in this PR and tested for both chat and completions and it works great)
3. Google Vertex API Gemini Model (this doesn't work yet and can
eventually be added to this codebase but we gotta add a new decoder for
the streaming responses of the gemini model through this API we can take
care of this later)

Sense of Urgency: This is a P0 because of enterprise requirements so I
would appreciate a fast approval and merging.

<!-- 💡 To write a useful PR description, make sure that your description
covers:
- WHAT this PR is changing:
    - How was it PREVIOUSLY.
    - How it will be from NOW on.
- WHY this PR is needed.
- CONTEXT, i.e. to which initiative, project or RFC it belongs.

The structure of the description doesn't matter as much as covering
these points, so use
your best judgement based on your context.
Learn how to write good pull request description:
https://www.notion.so/sourcegraph/Write-a-good-pull-request-description-610a7fd3e613496eb76f450db5a49b6e?pvs=4
-->

- Run this branch for Cody instance ->
https://github.com/sourcegraph/cody/pull/4606
- Ask @arafatkatze to dm you the siteadmin config to make things work
- Check the logs and play with completions and chat

<!-- All pull requests REQUIRE a test plan:
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->

<!--
1. Ensure your pull request title is formatted as: $type($domain): $what
3. Add bullet list items for each additional detail you want to cover
(see example below)
4. You can edit this after the pull request was merged, as long as
release shipping it hasn't been promoted to the public.
5. For more information, please see this how-to
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c?

Audience: TS/CSE > Customers > Teammates (in that order).

Cheat sheet: $type = chore|fix|feat $domain:
source|search|ci|release|plg|cody|local|...
-->

<!--
Example:

Title: fix(search): parse quotes with the appropriate context
Changelog section:

- When a quote is used with regexp pattern type, then ...
- Refactored underlying code.
-->

---------

Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Co-authored-by: Beatrix <beatrix@sourcegraph.com>
Co-authored-by: Stephen Gutekanst <stephen@sourcegraph.com>
(cherry picked from commit 1a6a7f7)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

backport 5.4.5099 label used to backport PRs to the 5.4.5099 release branch backports cla-signed failed-backport-to-5.4.5099 release-blocker Prevents us from releasing: https://about.sourcegraph.com/handbook/engineering/releases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants