Motivation

Add offline auto-tuning script for LoRA csgmv shrink / expand kernels (similar to MoE auto-tuning)

On H200 with Qwen3-Embedding-0.6B (rank=64), tuning yields 2-3x speedup on shrink kernels and 1.1-1.5x on expand kernels

Modifications

• Add lora_tuning_config.py: Config loader with LRU cache – same MoE loader
• Add tune_lora_csgmv.py: Offline script to generate configs
• Update kernel invocations to use the loaded configs (with existing defaults); maxnregs helped with occupancy so added that as kwargs
• Unit test for loading logic

python -m unittest test.manual.lora.test_lora_tuning_config -v

Fallback behavior

When no tuned config is found (no config file for the current GPU/model/Triton version), falls back to the original upstream defaults.

Usage

# Tune for a model (auto-derives all layer dims)
python benchmark/kernels/lora_csgmv/tune_lora_csgmv.py \
    --model Qwen/Qwen3-Embedding-0.6B --rank 64

# Configs saved to python/sglang/srt/lora/triton_ops/csgmv_configs/<triton_version>/
# Server automatically picks them up with --lora-backend csgmv

Accuracy Tests

No changes to kernel computation logic — only block size and launch params are tuned. The kernels produce identical outputs with different block sizes (verified by existing LoRA correctness tests).

Benchmarking and Profiling

Kernel-level tuning results (H200, Triton 3.5.1, Qwen3-Embedding-0.6B, rank=64)

Per-layer net savings at chunk_size=128 (shrink + expand combined)

E2E benchmark result

Launch Server

python -m sglang.launch_server         --tp-size 1     --model-path $BASE_MODEL        --is-embedding    --host 127.0.0.1        --port 30000             --disable-radix-cache --chunked-prefill-size -1 --max-prefill-tokens 32768       --enable-lora   --lora-paths v1=$UPDATED_ADAPTER  --lora-backend csgmv 

python -m sglang.bench_serving   --backend sglang-embedding   --host 127.0.0.1   --port 30000   --model $BASE_MODEL --dataset-name random  --random-input-len <num-tokens> --random-range-ratio 1.0  --num-prompts 120   --request-rate <rps>   --lora-name v1 

All % gains are relative to main @ chunk_size=16 (the current default behavior).

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.