[model] support MiniCPM-V 4.0#8747
Conversation
There was a problem hiding this comment.
Summary of Changes
Hello @tc-mb, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces support for the MiniCPM-V 4.0 model, expanding the framework's capabilities to include this new multimodal architecture. It integrates language and vision components and defines how modules are handled for optimization techniques like LoRA and quantization.
Highlights
- New Model Support: Added support for the MiniCPM-V 4.0 model by introducing a new
MiniCPMV4_0class, encapsulating the architecture and processing logic for this version. - LLM and Vision Module Integration: The
MiniCPMV4_0implementation integratesLlamaForCausalLMas its language model and utilizesIdefics2VisionTransformerwithResampler2_5for vision processing. - Multimodal Input Handling: Implemented methods for handling multimodal inputs, such as
get_vision_embedding,get_image_feature, andpad_input_idswithin theMiniCPMV4_0class. - Quantization and LoRA Mappings: Defined
packed_modules_mapping,supported_lora_modules, andbitsandbytes_stacked_params_mappingwithinMiniCPMV4_0to support LoRA fine-tuning and BitandBytes quantization.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request adds support for the MiniCPM-V 4.0 model. The implementation introduces a new class MiniCPMV4_0 which is similar to the existing MiniCPMV2_6. My review focuses on the maintainability and design of this new class. I've identified significant code duplication between the two version-specific classes and a violation of the Liskov Substitution Principle in the class hierarchy. I've provided detailed comments on these issues and suggested refactoring to improve the code quality.
|
|
||
| return pattern.pad_input_tokens(input_ids, image_inputs) | ||
|
|
||
| class MiniCPMV4_0(MiniCPMBaseModel): |
There was a problem hiding this comment.
The new class MiniCPMV4_0 duplicates code from MiniCPMV2_6. The following are identical:
- Class attributes:
packed_modules_mapping,supported_lora_modules,bitsandbytes_stacked_params_mapping,embedding_modules,embedding_padding_modules. - Methods:
init_vision_module,init_resampler,get_vision_embedding,get_image_feature,pad_input_ids.
This duplication reduces maintainability. Refactor common logic into a shared base class (e.g., MiniCPMBase) from which MiniCPMV2_6 and MiniCPMV4_0 inherit. Child classes would then implement version-specific logic, such as __init__ and init_llm.
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Could you use this model to replace the old minicpmv test? Should be under test/srt/test_vision_xxxxx |
JustinTong0323
left a comment
There was a problem hiding this comment.
Thanks for you contribution! May you resolve the comments first?
|
Your version of sglang appears to be outdated. It is recommended that you attempt to merge the main branch first. |
Your 3 revision suggestions have been modified. ^_^ |
Signed-off-by: tc-mb <caitianchi@modelbest.cn>
Signed-off-by: tc-mb <caitianchi@modelbest.cn> Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Motivation
Modifications
Accuracy Test
Benchmark & Profiling
Checklist