This repo is the official implementation of the ICCV-2023 paper: Cumulative Spatial Knowledge Distillation for Vision Transformers.

Environments:

• Python 3.6
• PyTorch 1.10.1
• torchvision 0.11.2

Install the package:

sudo pip3 install -r requirements.txt

• Download the dataset at https://image-net.org/ and put them to ./data/imagenet

python3 -m torch.distributed.launch --use_env --standalone --nnodes 1 --nproc_per_node 2 main.py --config configs/cskd_tiny.py --eval-only --ckpt {ckpt}

python3 -m torch.distributed.launch --use_env --standalone --nnodes 1 --nproc_per_node 8 main.py --config configs/cskd_tiny.py

If this repo is helpful for your research, please consider citing the paper:

@inproceedings{zhao2023cumulative,
  title={Cumulative Spatial Knowledge Distillation for Vision Transformers},
  author={Zhao, Borui and Song, Renjie and Liang, Jiajun},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={6146--6155},
  year={2023}
}

This repo is released under the MIT license. See LICENSE for details.

• Thanks for DeiT. We build this repo based on DeiT.