Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

Update Cody Bedrock support to use Anthropic messages API#61347

Merged
arafatkatze merged 6 commits into
mainfrom
cw/bedrock-messages-api
Mar 25, 2024
Merged

Update Cody Bedrock support to use Anthropic messages API#61347
arafatkatze merged 6 commits into
mainfrom
cw/bedrock-messages-api

Conversation

@chwarwick

@chwarwick chwarwick commented Mar 22, 2024

Copy link
Copy Markdown
Contributor

This updates the Bedrock provider to Anthropic's Messages API which includes support for the Claude 3 models.

Much of this is very close to duplication between the Anthropic and Bedrock providers, I considered consolidating but they are running different Anthropic api versions and imagine that they may diverge more in the future so the small amount of duplication does not feel overly burdensome.

Test plan

BYOK Setup (Claude 2 & Claude Instant 1.2)

With legacy request
streaming ✅

curl 'https://sourcegraph.test:3443/.api/completions/stream' \
-X POST \
-H 'authorization: token TOKEN' \
--data-raw '{"messages":[{"speaker":"human","text":"What is your name?"},{"speaker":"assistant"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":true}'

not streaming ✅

curl 'https://sourcegraph.test:3443/.api/completions/stream' \
-X POST \
-H 'authorization: token TOKEN' \
--data-raw '{"messages":[{"speaker":"human","text":"What is your name?"},{"speaker":"assistant"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":false}'

code-completions ✅

curl 'https://sourcegraph.test:3443/.api/completions/code' \
-X POST \
-H 'authorization: token TOKEN' \
--data-raw '{"messages":[{"speaker":"human","text":"What is your name?"},{"speaker":"assistant"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":true}'

With V1 request

streaming ✅

curl 'https://sourcegraph.test:3443/.api/completions/stream?api-version=1' \
-i \
-X POST \
-H 'authorization: token TOKEN' \
--data-raw '{"messages":[{"speaker":"system","text":"You only answer using emoji."},{"speaker":"human","text":"What is your name?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":true}'

non streaming ✅

curl 'https://sourcegraph.test:3443/.api/completions/stream?api-version=1' \
-X POST \
-H 'authorization: token TOKEN' \
--data-raw '{"messages":[{"speaker":"system","text":"You only answer using emoji."},{"speaker":"human","text":"What is your name?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":false}'

code-completions ✅

curl 'https://sourcegraph.test:3443/.api/completions/code?api-version=1' \
-X POST \
-H 'authorization: token TOKEN' \
--data-raw '{"messages":[{"speaker":"system","text":"You only answer using emoji."},{"speaker":"human","text":"What is your name?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":true}'

Using Claude 3

Chat
image

Autocomplete
image

Commands
image

@cla-bot cla-bot Bot added the cla-signed label Mar 22, 2024
@chwarwick chwarwick changed the title Updates Bedrock support to Anthropic Messages API including Claude 3 support Update Cody Bedrock support to use Anthropic messages API Mar 22, 2024
@chwarwick chwarwick requested review from a team and philipp-spiess March 22, 2024 20:03
philipp-spiess
philipp-spiess approved these changes Mar 25, 2024
View reviewed changes

@philipp-spiess philipp-spiess left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💅

Comment thread internal/completions/types/types.go Outdated

@philipp-spiess philipp-spiess Mar 25, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably also copy the tests over, grep for TestLegacyMessageConversion

@arafatkatze arafatkatze Mar 25, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@philipp-spiess Thats a good pointer and I covered it.

@arafatkatze arafatkatze force-pushed the cw/bedrock-messages-api branch from 815efaf to cd07216 Compare March 25, 2024 13:35
arafatkatze
arafatkatze approved these changes Mar 25, 2024
View reviewed changes

@arafatkatze arafatkatze left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Tested on different models of anthropic(both version 2 and 3) using AWS Bedrock
  • Fixed the test locations

@arafatkatze arafatkatze force-pushed the cw/bedrock-messages-api branch from cd07216 to 7b44a46 Compare March 25, 2024 13:52
@arafatkatze arafatkatze enabled auto-merge (squash) March 25, 2024 13:53
@arafatkatze arafatkatze merged commit 7ea54d4 into main Mar 25, 2024
@arafatkatze arafatkatze deleted the cw/bedrock-messages-api branch March 25, 2024 13:56
@arafatkatze arafatkatze mentioned this pull request Mar 26, 2024
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

2 more reviewers

@philipp-spiess philipp-spiess philipp-spiess approved these changes

@arafatkatze arafatkatze arafatkatze approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla-signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@chwarwick @philipp-spiess @arafatkatze