Motivation

1. Currently SGL expects LoRA weights to come in certain formats (e.g., q_proj, k_proj, v_proj) and conducts some stacking during loading time to meet expectation of the lora backends.

However, there are some models (e.g., Phi4MM whose weight tensors come in pre-fused format, which is not supported as-is by SGL today:
.

2. The current LoRA impl in SGL uses layer name and operation name (e.g., "qkv_proj") to uniquely identify LoRA adapters, which works perfectly fine for conventional LLM. However, for model architectures that have more than one attention stack (e.g., VLM), we need more accurate mapping between LoRA weights and the base model modules.

Modifications

1. Generalize the preprocessing function from stacking-only to "normalization" at both directions (stacking, splitting, replicating, etc.) based on the initial tensor shape.
2. To limit the scope of this PR, I did not fully implement the accurate mapping, but only introduced a map_lora_module_name that right now only serves as a "filter" to ensure LoRAManager does not incorrectly maps LoRA weights to unwanted modules (e.g., vision towers).

Example

In my local branch, I can verify that Phi4MM LoRA can be loaded successfully and observed a significant increase in MMMU score from 0.38 to 0.472.

It's worth noting that, I noticed that the MMMU is still lower than what's claimed by the author (0.55). However, it's difficult to conclude the source of the discrepancy, as I am seeing similar situation for exisitng non-LoRA models as well. The root cause could be one of the following: (1) incorrect model implementation (2) issues with the benchmark script (3) paper authors had a different benchmarking setup. I will add it to my follow-up list.

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.