Skip to content

Large Multimodal Models in AgentChat#554

Merged
sonichi merged 18 commits into
microsoft:mainfrom
BeibinLi:lmm
Nov 6, 2023
Merged

Large Multimodal Models in AgentChat#554
sonichi merged 18 commits into
microsoft:mainfrom
BeibinLi:lmm

Conversation

@BeibinLi

@BeibinLi BeibinLi commented Nov 5, 2023

Copy link
Copy Markdown
Contributor

Why are these changes needed?

Related issue number

Checks

@sonichi

sonichi commented Nov 5, 2023

Copy link
Copy Markdown
Contributor

Could you resolve the conflicts?

@codecov-commenter

codecov-commenter commented Nov 5, 2023

Copy link
Copy Markdown

Codecov Report

Merging #554 (8d2e64b) into main (306ac4d) will increase coverage by 12.91%.
The diff coverage is 66.96%.

@@             Coverage Diff             @@
##             main     #554       +/-   ##
===========================================
+ Coverage   32.24%   45.15%   +12.91%     
===========================================
  Files          22       25        +3     
  Lines        2785     3005      +220     
  Branches      634      707       +73     
===========================================
+ Hits          898     1357      +459     
+ Misses       1820     1525      -295     
- Partials       67      123       +56     
Flag Coverage Δ
unittests 44.92% <66.96%> (+12.75%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
autogen/agentchat/__init__.py 100.00% <100.00%> (ø)
autogen/code_utils.py 66.94% <100.00%> (+18.45%) ⬆️
autogen/img_utils.py 77.21% <77.21%> (ø)
.../agentchat/contrib/multimodal_conversable_agent.py 55.31% <55.31%> (ø)
autogen/agentchat/contrib/llava_agent.py 56.62% <56.62%> (ø)

... and 12 files with indirect coverage changes

@BeibinLi

BeibinLi commented Nov 6, 2023

Copy link
Copy Markdown
Contributor Author

Could you resolve the conflicts?
Resolved by rebasing to main.

So, it can still handle human_input_mode
and max_consecutive_reply
Comment thread .github/workflows/contrib-lmm.yml Outdated
Comment thread autogen/agentchat/contrib/multimodal_conversable_agent.py
Comment thread autogen/agentchat/contrib/multimodal_conversable_agent.py Outdated
Comment thread autogen/agentchat/contrib/multimodal_conversable_agent.py Outdated
Comment thread website/blog/2023-11-06-LMM-Agent/index.mdx Outdated
Comment thread website/docs/Installation.md Outdated
Comment thread .github/workflows/contrib-lmm.yml Outdated
Fixing: typos, blogs, yml, and add OpenAIWrapper
1. Add a notebook for GPT-4V. LLava notebook also updated.
2. img_utils updated
3. GPT-4V formatter now return base64 image with mime type
4. Infer mime type directly from b64 image content (while loading
   without suffix)
5. Test cases modified according to all the related changes.

@sonichi sonichi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving for a quick release of beta version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants