Skip to content

Dev/v0.2 - Enable streaming support for openai v1.0.0b3#491

Closed
Alvaromah wants to merge 3 commits into
microsoft:dev/v0.2from
Alvaromah:dev/v0.2-streaming
Closed

Dev/v0.2 - Enable streaming support for openai v1.0.0b3#491
Alvaromah wants to merge 3 commits into
microsoft:dev/v0.2from
Alvaromah:dev/v0.2-streaming

Conversation

@Alvaromah

@Alvaromah Alvaromah commented Oct 30, 2023

Copy link
Copy Markdown
Contributor

Why are these changes needed?

The OpenAI API, along with other LLM frameworks, offers streaming capabilities that enhance debugging workflows by eliminating the need to wait for complete responses, resulting in a more efficient and time-saving process.

This is a simple mechanism to support streaming.
Tested on openai v1.0.0b3.

To enable streaming just use this code:

llm_config={
    "config_list": config_list,
    # Enable, disable streaming (defaults to False)
    "stream": True,
}

assistant = autogen.AssistantAgent(
    name="assistant",
    llm_config=llm_config,
)

Related issue number

Related to #465, #217

Checks

@Alvaromah Alvaromah mentioned this pull request Oct 30, 2023
Closed
3 tasks
sonichi
sonichi reviewed Oct 31, 2023
View reviewed changes

@sonichi sonichi left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me find a few reviewers who are interested in streaming support. Feel free to invite if you know any.
Meanwhile, the structure looks good to me. It's a good time to add a test.

Comment thread autogen/oai/client.py
else:
# If streaming is not enabled, send a regular chat completion request
# Ensure streaming is disabled
params["stream"] = False

@sonichi sonichi Oct 31, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed? I'd like to avoid modifying the original dict if not necessary.

Comment thread autogen/oai/client.py
@ragyabraham

ragyabraham commented Nov 1, 2023

Copy link
Copy Markdown
Contributor

This works great for me. We've also managed to integrate it with #394 (see gif below).

streaming with sockets

Given that streaming is essentially a UX centric issue, it would probably make sense to open another PR to formally integrate streaming with some sort of messaging framework (e.g. sockets).

ragyabraham
ragyabraham approved these changes Nov 1, 2023
View reviewed changes

@ragyabraham ragyabraham left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pq-dong

pq-dong commented Nov 2, 2023

Copy link
Copy Markdown

Are there any plans to merge into microsoft:dev/v0.2？
When to be merged into microsoft:dev/v0.2 ?

@sonichi sonichi requested review from a team and pcdeadeasy November 2, 2023 03:42
@sonichi

sonichi commented Nov 2, 2023

Copy link
Copy Markdown
Contributor

Are there any plans to merge into microsoft:dev/v0.2？ When to be merged into microsoft:dev/v0.2 ?

After the conversation is resolved, and no further revision request from other reviewers.

@franciscoabenza

franciscoabenza commented Nov 2, 2023 •
edited
Loading

Copy link
Copy Markdown

@ragyabraham 🙏 Please please please, can you explain me how are you running the UI?/ is it part of your autogen branch?

streaming with sockets

@ragyabraham

ragyabraham commented Nov 2, 2023

Copy link
Copy Markdown
Contributor

Hey @franciscoabenza yeah, this a web UI we are building for autogen. You can find it here

@sonichi sonichi deleted the branch microsoft:dev/v0.2 November 4, 2023 04:33
@sonichi sonichi closed this Nov 4, 2023
@sonichi

sonichi commented Nov 5, 2023

Copy link
Copy Markdown
Contributor

dev/v0.2 is merged into main. Could you recreate the PR to target main? @Alvaromah

@Alvaromah

Alvaromah commented Nov 7, 2023

Copy link
Copy Markdown
Contributor Author

dev/v0.2 is merged into main. Could you recreate the PR to target main? @Alvaromah

Sure! I will, as soon as possible.

@sonichi sonichi mentioned this pull request Nov 7, 2023
Closed
@Alvaromah

Alvaromah commented Nov 8, 2023

Copy link
Copy Markdown
Contributor Author

dev/v0.2 is merged into main. Could you recreate the PR to target main? @Alvaromah

Created PR #597 to target main.

@hughlv hughlv mentioned this pull request Dec 7, 2023
Open
@anu43

anu43 commented Jan 18, 2025

Copy link
Copy Markdown

I came upon this by chance. It is huge tbh. Is it present in the documentation? I couldn't find such a capability anywhere.

@ekzhu

ekzhu commented Jan 18, 2025

Copy link
Copy Markdown
Contributor

I came upon this by chance. It is huge tbh. Is it present in the documentation? I couldn't find such a capability anywhere.

@anu43, streaming has been available since v0.2.

https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.models.openai.html#autogen_ext.models.openai.BaseOpenAIChatCompletionClient.create_stream

https://microsoft.github.io/autogen/stable/user-guide/core-user-guide/framework/model-clients.html#streaming-response

@anu43

anu43 commented Jan 21, 2025

Copy link
Copy Markdown

I also got streamed output in v0.2 by adding stream: True in the LLM config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxim-saplin maxim-saplin Awaiting requested review from maxim-saplin

@victordibia victordibia Awaiting requested review from victordibia

@wayliums wayliums Awaiting requested review from wayliums

@AaronWard AaronWard Awaiting requested review from AaronWard

@pcdeadeasy pcdeadeasy Awaiting requested review from pcdeadeasy

3 more reviewers

@sonichi sonichi sonichi left review comments

@franciscoabenza franciscoabenza franciscoabenza left review comments

@ragyabraham ragyabraham ragyabraham approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Alvaromah @ragyabraham @pq-dong @sonichi @franciscoabenza @anu43 @ekzhu