[Model][LoRA]LoRA support added for Qwen2VLForConditionalGeneration#10022
[Model][LoRA]LoRA support added for Qwen2VLForConditionalGeneration#10022DarkLight1337 merged 8 commits intovllm-project:mainfrom
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
There was a problem hiding this comment.
Although vllm currently only supports lora inference for language model, considering that inference for vision modules will be supported in the future, we should also add vision encoder related modules.
There was a problem hiding this comment.
However, currently fine-tuning only the language model with LoRA achieves good results and meets the needs of most users. Further optimizations can be made later if necessary.
There was a problem hiding this comment.
Yes, I agree with your view. I just think it might be better to complete the additions first. If you're interested, you can refer to Qwen-VL's implementation, see: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/qwen.py#L1015-L1046.
There was a problem hiding this comment.
Thanks, I'll take a look.
|
You can run |
Signed-off-by: ericperfect <ericperfectttt@gmail.com>
Signed-off-by: ericperfect <ericperfectttt@gmail.com>
Signed-off-by: ericperfect <ericperfectttt@gmail.com>
Signed-off-by: ericperfect <ericperfectttt@gmail.com>
Signed-off-by: ericperfect <ericperfectttt@gmail.com>
Signed-off-by: ericperfect <ericperfectttt@gmail.com>
Signed-off-by: ericperfect <ericperfectttt@gmail.com>
jeejeelee
left a comment
There was a problem hiding this comment.
Alternatively, you could add TODO above the supported_lora_modules , indicating that we need to add LoRA for the visual encoder in the future. Additionally, you need to update the documentation to specify that this model supports LoRA.
…sage and update documentation. Signed-off-by: ericperfect <ericperfectttt@gmail.com>
|
Thanks, I already have added the TODO information and updated the documentation. |
DarkLight1337
left a comment
There was a problem hiding this comment.
Thanks for adding this!
Also thanks @jeejeelee for the review
…llm-project#10022) Signed-off-by: ericperfect <ericperfectttt@gmail.com> Signed-off-by: Loc Huynh <jc1da.3011@gmail.com>
…llm-project#10022) Signed-off-by: ericperfect <ericperfectttt@gmail.com> Signed-off-by: Sumit Dubey <sumit.dubey2@ibm.com>
…llm-project#10022) Signed-off-by: ericperfect <ericperfectttt@gmail.com>
…llm-project#10022) Signed-off-by: ericperfect <ericperfectttt@gmail.com> Signed-off-by: LeiWang1999 <leiwang1999@outlook.com>
[Model][LoRA]LoRA support added for Qwen2VLForConditionalGeneration
LoRA supported for inference in Qwen2VL
ping @ywang96 @DarkLight1337