Arxiv | OpenReview

MinKyu Lee, Sangeek Hyun, Woojin Jun, Hyunjun Kim, Jiwoo Chung, Jae-Pil Heo*
Sungkyunkwan University
*: Corresponding Author

This work analyzes the training dynamics of Image Restoration (IR) Transformers and uncovers a critical yet overlooked issue: conventional LayerNorm (LN) drives feature magnitudes to diverge to a million scale and collapses channel-wise entropy. We analyze this in the perspective of networks attempting to bypass LN's constraints that conflict with IR tasks. Accordingly, we address two misalignments between LN and IR: 1) per-token normalization disrupts spatial correlations, and 2) input-independent scaling discards input-specific statistics. To address this, we propose Image Restoration Transformer Tailored Layer Normalization i-LN, a simple drop-in replacement that normalizes features holistically and adaptively rescales them per input. We provide theoretical insights and empirical evidence that this simple design effectively leads to both improved training dynamics and thereby improved performance, validated by extensive experiments.

• ✅ Code release
• ✅ Model checkpoint release

cd iLN
bash _custom_setup.sh

• Checkpoints can be downloaded from here.
• Datasets can be downloaded and preprocessed from BasicSR.

• HAT-mini: referred to as HAT_1 in the paper; smaller than HAT-S
• HAT-dagger: the full-sized HAT model

cd iLN
python basicsr/train.py -opt options/train/HAT-mini/SRx2_HAT-mini_baseline.yml  # modify the ckpt/dataset path as required
python basicsr/train.py -opt options/train/HAT-mini/SRx4_HAT-mini_baseline.yml

cd iLN
python basicsr/train.py -opt options/train/HAT-mini/SRx2_HAT-mini_iLN.yml
python basicsr/train.py -opt options/train/HAT-mini/SRx4_HAT-mini_iLN.yml

cd iLN
python basicsr/train.py -opt options/train/HAT-dagger/SRx2_HAT-dagger_iLN.yml
python basicsr/train.py -opt options/train/HAT-dagger/SRx4_HAT-dagger_iLN.yml

cd iLN
python basicsr/test.py -opt options/test/HAT-mini/SRx2_HAT-mini_baseline.yml  # modify the ckpt/dataset path as required
python basicsr/test.py -opt options/test/HAT-mini/SRx4_HAT-mini_baseline.yml

cd iLN
python basicsr/test.py -opt options/test/HAT-mini/SRx2_HAT-mini_iLN.yml
python basicsr/test.py -opt options/test/HAT-mini/SRx4_HAT-mini_iLN.yml

cd iLN
python basicsr/test.py -opt options/test/HAT-dagger/SRx2_HAT-dagger_iLN.yml
python basicsr/test.py -opt options/test/HAT-dagger/SRx4_HAT-dagger_iLN.yml

This project is built based on:

• BasicSR
• SwinIR
• HAT
• DRCT

Please contact me via 2minkyulee@gmail.com for any inquiries.

@article{lee2025analyzing,
  title={Analyzing the Training Dynamics of Image Restoration Transformers: A Revisit to Layer Normalization},
  author={Lee, MinKyu and Hyun, Sangeek and Jun, Woojin and Kim, Hyunjun and Chung, Jiwoo and Heo, Jae-Pil},
  journal={arXiv preprint arXiv:2504.06629},
  year={2025}
}