Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledge (AAAI 2026)

Download the repo and install dependencies.

cd LoRA_Null
pip install -r requirements.txt

sh step1.sh

CUDA_VISIBLE_DEVICES=0 python build_corda.py \
    --model_id "meta-llama/Llama-2-7b-hf" \
    --singular_aware \
    --r {rank} \
    --use_cache \
    --calib_dataset "nqopen" \
    --calib_loader_size 256 \
    --save_model \
    --save_path {path_to_decomposed_model}

Arguments:

• --model_id is the pre-trained model for decomposition.
• --singular_aware adopts our LoRA-Null inilization.
• --r is the low rank of LoRA, e.g. 128.
• --use_cache adopts the dataloader and covariance matrices saved in Adapter/cache, to avoid calculating the covariance matrices again.
• --calib_dataset specifies the dataset to sample data to obtain covariance matrices. We use QA datasets "nqopen".
• --calib_loader_size is the number of sampled data.
• --save_model saves the initialized model in --save_path.

sh step2.sh

After training, LoRA adapter can be merged with the base model by runing: sh step3.sh

Inference on world knowledge benchmarks is based on EleutherAI/lm-evaluation-harness. For example, we evaluate by: sh step4.sh

accelerate launch -m lm_eval \
    --model hf \
    --model_args pretrained={path_to_merged_model},trust_remote_code=True,dtype=float16 \
    --output_path {result_path}/nq_open.json \
    --tasks nq_open,triviaqa,nq_open \
    --batch_size auto \
    --max_batch_size 8 \
    --device cuda

Inference on Math:

Evaluation on Gsm8k and Math can be performed by: sh step5.sh

sh tools/inference_Math.sh {path_to_merged_model}

Inference on Code and Instruction Following:

Evaluation on HumanEval and MBPP is based on bigcode-evaluation-harness. Evaluation on MTBench is based on FastChat. We use their default settings for evaluation.

Our codes are modified from https://github.com/iboing/CorDA