Using sockets rather than print by joshkyh · Pull Request #1290 · microsoft/autogen

joshkyh · 2024-01-16T21:29:25Z

Reviewers:
@ekzhu on navigating the refactoring efforts in #1240
@victordibia regarding the generalized approach in #394 (comment)

Why are these changes needed?

This PR allows the output of Agents like ConversableAgent to be emitted through socket rather than print to console, allowing integration with other front end tools that could sit outside of Autogen. This PR is co-authored by @ragyabraham and myself. Tagging @tomlynchRNA for awareness.

Related issue number

Closes #394
Associated #1199 for emitting to Teams front end. @tyler-suard-parker
Dependent on #1240 for cleaner/easier implementation.

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

codecov-commenter · 2024-01-16T21:30:38Z

Codecov Report

Attention: Patch coverage is 54.76190% with 19 lines in your changes missing coverage. Please review.

Project coverage is 41.92%. Comparing base (260e0cf) to head (e9af709).
Report is 1697 commits behind head on main.

Files with missing lines	Patch %	Lines
autogen/agentchat/contrib/stream_handler.py	60.00%	10 Missing ⚠️
autogen/agentchat/conversable_agent.py	30.00%	6 Missing and 1 partial ⚠️
autogen/agentchat/groupchat.py	71.42%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1290      +/-   ##
==========================================
+ Coverage   31.98%   41.92%   +9.93%     
==========================================
  Files          33       34       +1     
  Lines        4415     4456      +41     
  Branches     1030     1095      +65     
==========================================
+ Hits         1412     1868     +456     
+ Misses       2887     2437     -450     
- Partials      116      151      +35

Flag	Coverage Δ
unittests	`41.85% <54.76%> (+9.91%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ekzhu · 2024-01-16T21:46:36Z

This is a great addition. I am trying to get a good understanding of the design.

Is this aiming to move agent-to-agent communication on to websocket/streaming? There is a parallel effort on this front from Add RemoteAgent and Receiver #1289
To handle printing to a stream, could this be done with a special logging middleware as proposed in the middleware design in Refactorization of ConversableAgent to unify async and sync code and better extensibility #1240?

joshkyh · 2024-01-16T22:09:54Z

This is a great addition. I am trying to get a good understanding of the design.

Is this aiming to move agent-to-agent communication on to websocket/streaming? There is a parallel effort on this front from Add RemoteAgent and Receiver #1289

Thanks! I'm glad you're excited by it too.

@ragyabraham please feel free to add.

I think the fundamental difference to #1289 is that here, we're not trying to move the agent-to-agent message transmission to socket, nor have an Agent hosted elsewhere. If my understanding is correct
jacks_cal = RemoteAgent("jacks_cal", host="localhost", port=45554)
means that jacks_cal can be hosted at another IP address. This PR is not concerned with that. Instead, this PR is about emitting messages of what each Agent said onto the socket, so that another front-end tool can display it and not for the purpose of sending it to another agent. In our use case, the front-end is a web browser, there seems to be another use case of displaying it in Teams in #1199.

To handle printing to a stream, could this be done with a special logging middleware as proposed in the middleware design in Refactorization study with design patterns (was refactorization of hooks) #1240?

I think so, however, I'm not very familiar with hooks, and would need to work with you or @davorrunje to iterate on how to implement it properly through the reviews. If there's any tips anyone can share before we start developing, that might help reduce reviews/rework! :)

ekzhu · 2024-01-16T22:38:17Z

need to work with you or @davorrunje to iterate on how to implement it properly through the reviews

Certainly! I am here to make sure us don't have to redo a lot of work. I am just thinking if #1240 go through first, it will be much easier for us to implement something like logging and emitting to a frontend.

gagb · 2024-01-21T06:02:56Z

+        use_agent_stream: Optional[bool] = False,
+        get_socket_client_function: Optional[Callable] = None,
+        sid: Optional[str] = None,


Add docstring for new arguments.
Consider choosing more descriptive name than sid?

davorrunje · 2024-01-21T10:19:27Z

        llm_config: Optional[Union[Dict, Literal[False]]] = None,
        default_auto_reply: Optional[Union[str, Dict, None]] = "",
        description: Optional[str] = None,
+        use_agent_stream: Optional[bool] = False,


I can easily imagine more than just two types of output stream because there are so many different ways to stream data on the internet. Why don't we define AgentOutput protocol for outputting data and implement two initial classes: PrintAgentOutput and SocketAgentOutput implementing the protocol? That way we would be extensible in future without being forced to change the ConversableClient class.

+1 On the suggestion.

In this approach, should we plan for mutually exclusive outputs or a list of outputs? E.g. both print and socket outputs happening.

ekzhu · 2024-01-21T21:22:48Z

+
+        if self.use_agent_stream:
+            # Generate sid if None
+            if self.sid is None:


These implementation details can be refactored out to a separate delegate class to avoid bloating the ConversableAgent.

ekzhu · 2024-01-21T21:23:18Z

            )

+        if use_agent_stream:
+            # Generate sid if None


Refactor out to delegate to avoid bloating

joshkyh · 2024-01-21T23:27:05Z

@ekzhu @victordibia
@ragyabraham and I discussed this during our stand-up. We have to withdraw working on this due to other priorities. Can I assign this to the MSFT team?

ekzhu · 2024-01-23T16:58:34Z

@joshkyh no worries. I just unassigned you from the PR.

Tylersuard · 2024-01-27T22:19:17Z

@ekzhu can we have some simple example code for how to implement this on our own?

Tylersuard · 2024-01-27T22:20:26Z

+                # Upper, digits. 5chars - 5chars - 5chars
+                candidates = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
+                sid = "-".join(
+                    [


Why are we choosing random letters here? Could we just use uuid?

Tylersuard

Please add example code for how to use streaming with agents. This is a really cool feature, and I'm sure a lot of people will want to know how to use it.

Tylersuard · 2024-01-27T22:22:04Z

+                sid = "-".join(
+                    [
+                        "".join(random.choices(candidates, k=5)),
+                        "".join(random.choices(candidates, k=5)),


Again, why are we using random letters here instead of using uuid?

Tylersuard · 2024-01-27T22:23:30Z

+              autogen.agentchat.Agent(name="Agent2"), 
+              autogen.agentchat.Agent(name="Agent3")]
+
+    with pytest.raises(ValueError, match="get_socket_client_function is required if use_agent_stream is True"):


Can we also have a test that assures streaming is working?
Also, some example code which instantiates an agent and sets up socket streaming would be helpful.

ekzhu · 2024-01-28T04:21:44Z

@Tylersuard Thanks for your interest. This is surely a feature we hope to have but don't have cycle to work on. If you are interested, you can take a look at it. A good place to start is in the conversable_agent.py file, in the method generate_reply. Before the reply message gets sent out you can intercept there and add an async write to a stream, which can be passed in as part of the constructor.

#1240 will refactor ConversableAgent to make this type of extension very easy to do. i.e., it will be a middleware class that you put inside ConversableAgent. You can also choose to work from that branch.

bitnom · 2024-02-14T01:50:52Z

I have implemented this sort of thing in autogen multiple times so far. IMO the best 'official' way to handle it would be to introduce an input Queue and an output Queue, allowing the user to specify and/or override them.

ekzhu · 2024-02-14T16:17:14Z

@joshkyh shall we close this one for now? The code base has moved a lot since.

ekzhu · 2024-02-14T16:18:13Z

I have implemented this sort of thing in autogen multiple times so far. IMO the best 'official' way to handle it would be to introduce an input Queue and an output Queue, allowing the user to specify and/or override them.

Great, you are welcome to go to #1551 to check it out. And there will be follow-up efforts on how to integrate it with frontend.

Params for conversableAgent and GroupChat Manager

b68fb59

joshkyh had a problem deploying to openai1 January 16, 2024 21:29 — with GitHub Actions Failure

joshkyh self-assigned this Jan 16, 2024

joshkyh had a problem deploying to openai1 January 16, 2024 21:29 — with GitHub Actions Failure

joshkyh assigned ragyabraham Jan 16, 2024

joshkyh requested review from ekzhu and victordibia January 16, 2024 21:29

joshkyh added dev labels Jan 16, 2024

Merge branch 'main' into use-sockets

4559c92

joshkyh had a problem deploying to openai1 January 16, 2024 21:36 — with GitHub Actions Failure

ekzhu mentioned this pull request Jan 16, 2024

Support for WebSockets, streaming responses to a frontend [Feature Request]: #1199

Closed

joshkyh had a problem deploying to openai1 January 18, 2024 00:07 — with GitHub Actions Failure

victordibia mentioned this pull request Jan 19, 2024

[P2] Stream messages from agents to UI as they are generated #1344

Closed

gagb reviewed Jan 21, 2024

View reviewed changes

davorrunje reviewed Jan 21, 2024

View reviewed changes

ChristianWeyer mentioned this pull request Jan 21, 2024

Agent output streams crewAIInc/crewAI#169

Closed

ekzhu reviewed Jan 21, 2024

View reviewed changes

victordibia mentioned this pull request Jan 21, 2024

User input does not show in the UI , instead one has to switch back to terminal. AutogenStudio #1278

Closed

ekzhu unassigned joshkyh and ragyabraham Jan 23, 2024

sonichi requested a review from Tylersuard January 27, 2024 16:53

Tylersuard reviewed Jan 27, 2024

View reviewed changes

Tylersuard suggested changes Jan 27, 2024

View reviewed changes

davorrunje mentioned this pull request Feb 5, 2024

Introducing IOStream protocol and adding support for websockets #1551

Merged

11 tasks

joshkyh closed this Feb 14, 2024

joshkyh deleted the use-sockets branch February 14, 2024 20:01

Conversation

joshkyh commented Jan 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

codecov-commenter commented Jan 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ekzhu commented Jan 16, 2024

Uh oh!

joshkyh commented Jan 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ekzhu commented Jan 16, 2024

Uh oh!

gagb Jan 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshkyh commented Jan 21, 2024

Uh oh!

ekzhu commented Jan 23, 2024

Uh oh!

Tylersuard commented Jan 27, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tylersuard left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ekzhu commented Jan 28, 2024

Uh oh!

bitnom commented Feb 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ekzhu commented Feb 14, 2024

Uh oh!

ekzhu commented Feb 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

joshkyh commented Jan 16, 2024 •

edited

Loading

codecov-commenter commented Jan 16, 2024 •

edited

Loading

joshkyh commented Jan 16, 2024 •

edited

Loading

gagb Jan 21, 2024 •

edited

Loading

Tylersuard left a comment •

edited

Loading

bitnom commented Feb 14, 2024 •

edited

Loading