Custom Model Client support#1345
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #1345 +/- ##
===========================================
+ Coverage 33.12% 46.59% +13.47%
===========================================
Files 42 42
Lines 5051 5112 +61
Branches 1157 1239 +82
===========================================
+ Hits 1673 2382 +709
+ Misses 3250 2530 -720
- Partials 128 200 +72
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
sonichi
left a comment
There was a problem hiding this comment.
Looks good. I think a few places in the doc website need to be updated. They can be done in a separate PR:
https://microsoft.github.io/autogen/docs/FAQ#can-i-use-non-openai-models
https://microsoft.github.io/autogen/docs/Use-Cases/enhanced_inference#api-unification
https://microsoft.github.io/autogen/docs/Use-Cases/enhanced_inference#error-handling
https://microsoft.github.io/autogen/docs/llm_endpoint_configuration
Co-authored-by: Chi Wang <wang.chi@microsoft.com>
|
All the tests are passed except GPTAssistantAgent: https://github.com/microsoft/autogen/actions/runs/7733107408/job/21084512475?pr=1345 |
davorrunje
left a comment
There was a problem hiding this comment.
Overall, this is great! I have a few stylistic suggestions to make it a bit more pythonic.
|
@olgavrou I fixed the failing tests by introducing an additional protocol You can see my changes here: If you wish to merge it with your branch: There are a few todos left in the code to make type checks and autocomplete tighter, but functionally this is it. |
added a fix for it, it needs access to the internal client that was now refactored to belong to the |
|
Just want to say I am so happy it is merged finally! |
|
Fantastic work, thank you |
Why are these changes needed?
This PR adds support for custom client calls to be made. The idea is that the user can specify their own custom client class and load it into the configuration (e.g. for loading local models). As long as they adhere to the
Clientclass's interface and response protocol then everything should work fine with only this addition to the config (full usage shown in the unit test):So if we have the class
CustomClientthe class name (as a string) needs to be set as the api_type in the config:Then once an agent is created the custom client can be registered:
agent.register_custom_client(CustomClient, [optional other args for the constuctor of CustomClient])Notes:
Ideally the client interface would not live under a directory called
oaiand the client wrapper wouldn't be calledOpenAIWrapperbut something more generic, but that can be examined at a later dateRelated PR
Replaces #831 and has less refactoring done so that the change is more contained
Checks