This repo hosts PoLAR, a polar-decomposition-inspired reparameterization for fine-tuning LLMs in a parameter-efficient fashion. PoLAR resolves the rank inefficiency encountered in LoRA and boosts performance on downstream tasks for model sizes up to and including 30B. The paper is available at https://arxiv.org/abs/2506.03133.

LoRA often does not fully exploit the low-rank subspace, leading to a low effective (stable) rank. In particular, even if the rank of LoRA is chosen as r=32 or r=16, the stable rank of $\Delta W = BA$ is often close to 1; see an example of applying LoRA to LLaMA2-7B in the figure below. Informally, LoRA leverages an approximately rank-1 subspace.

This motivating finding occurs across several datasets and a minimal reproducible example of it is given in the notebooks/lora_low_effective_rank.ipynb notebook which evaluates the stable rank of the official LoRA DeBERTa XXL checkpoints.

PoLAR is a polar-decomposition-inspired reparameterization of LoRA. In particular, we introduce direction matrices $X$ and $Y$ with orthonormal columns (i.e., $X$ and $Y$ live in Stiefel manifolds), and an unconstrained scale matrix $\Theta$. The low-rank update is then formulated as $\Delta W = X \Theta Y^\top$. Matrices $X$ and $Y$ are optimized using tools from Riemannian optimization.

Such a co-design of architecture and optimizer significantly improves the stable rank in low-rank adaptation. This is illustrated in the figure above: PoLAR's stable rank varies with r and increases throughout training.

Dependencies are given in requirements.txt. To recreate the environment, please run

cd path/to/polar/repo
conda create --name polar python=3.10
conda activate polar
pip install -r requirements.txt

Note that the Gemma-3 experiments require newer versions. For the setup, see the section on Gemma-3.

To reproduce, please run

bash scripts/train_and_eval.sh <adapter_name> <lr> <reg_lambda> <num_epochs> <rank> <dataset>

To run the evaluation of checkpoints, please run

bash scripts/eval.sh /path/to/adapter/dir <task> meta-llama/Llama-2-7b-hf

For instance, to run PoLAR, LoRA, and DoRA on PIQA with rank 32

bash scripts/train_and_eval.sh polar 4e-3 5e-3 5 32 piqa 
bash scripts/train_and_eval.sh lora 4e-4 5e-3 5 32 piqa 
bash scripts/train_and_eval.sh dora 4e-4 5e-3 5 32 piqa 

The experiments with Gemma-3 require newer versions and the environment builds on top of nvcr.io/nvidia/pytorch:25.03-py3. To set up the container and run the evaluation of Gemma-3 checkpoints, please run:

# set up
export HF_TOKEN=your_hf_token
cd /path/to/polar/repo
docker build --file gemma3.Dockerfile -t polar:dev .
docker run --rm -it --gpus '"device=1,2,3,4"' -e HF_TOKEN=$HF_TOKEN polar:dev

# in container
bash scripts/eval_docker.sh /path/to/adapter/dir gsm8k google/gemma-3-27b-pt

# remove container
docker rmi polar:dev
docker system prune -a

For computation of the stable rank of the official LoRA checkpoints available at https://github.com/microsoft/LoRA, please see lora_low_effective_rank.ipynb. The notebook runs easily on google colab.

To reproduce the stable rank and pairwise distance plot of our trained adapters, please run the following after having trained any of the adapters above.

python stable_rank/analyze_sr.py --peft_model_id=/path/to/ckpt_dir --save_path=/path/to/ckpt_dir

Please run runtime_comparison.ipynb.

Note that the information about the custom module mapping does not persist when you save the PEFT model. When loading the model, you have to register the custom modules again:

from peft import PeftModel, PeftConfig
from peft.tuners.polar.layer import PoLARLinear

peft_config = PeftConfig.from_pretrained(peft)
custom_module_mapping = {torch.nn.Linear: PoLARLinear}
# register the new mapping
peft_config._register_custom_module(custom_module_mapping)
model = PeftModel.from_pretrained(self._model, peft, revision=revision, config=peft_config)

We require an editable install of lm-evaluation-harness due to the following:

• Registration of custom module mapping in lm_eval/models/huggingface.py. See discussion above.
• Avoidance of vocab_size check for Gemma 3 which raises AttributeError. See issue huggingface/transformers#36683.
• Adapting doc_to_text in tasks/gsm8k/gsm8k.yaml and tasks/hendrycks_math/hendrycks_math.yaml.

• The registration of the parameterization was used for research purposes only and can be disabled in practical scenarios to increase runtime efficiency. The appropriate line to comment out is line 119 in peft/src/peft/tuners/polar/layer.py.

• The finetune_commonsense.py script is originally adapted from https://github.com/tloen/alpaca-lora/blob/main/finetune.py.
• The lm-evaluation-harness is taken from https://github.com/EleutherAI/lm-evaluation-harness.
• The editable PEFT version is based on v0.14.1.dev0 of https://github.com/huggingface/peft.
• The run_glue.py script is adapted from https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py.

@misc{lion_polar_2025,
	title = {{PoLAR}: {Polar}-{Decomposed} {Low}-{Rank} {Adapter} {Representation}},
	shorttitle = {{PoLAR}},
	url = {https://arxiv.org/abs/2506.03133},
	journal = {arXiv.org},
	author = {Lion, Kai and Zhang, Liang and Li, Bingcong and He, Niao},
	year = {2025},
}