Wenzheng Zeng¹, Yang Xiao¹, Sicheng Wei¹, Jinfang Gan¹, Xintao Zhang¹, Zhiguo Cao¹, Zhiwen Fang²,
Joey Tianyi Zhou³

¹Huazhong University of Science and Technology, ²Southern Medical University, ³A*STAR

• [2026.4] We release MPEblink2.0 and InstBlink++, check out our extended report for details. Besides, also check out our new work DeFB, featured with higher eyeblink detection performance and easy-to-use environment (without MMCV dependency).
• [2023.6] We release MPEblink and Instblink.
• [2023.2] Our work is accepted by CVPR 2023.

• New Task: It is the first time that the task of instance-level multi-person eyeblink detection in untrimmed videos is formally defined and explored. We think that a good multi-person eyeblink detection algorithm should be able to (1) detect and track human instances’ faces reliably to ensure the instance-level analysis ability along the whole video, and (2) detect eyeblink boundaries accurately within each human instance to ensure the precise awareness of their eyeblink behaviors. We design new metrics to give attention to both instance awareness quality and eyeblink detection quality;
• New Dataset: To support this research task, we introduce MPEblink 1.0 & MPEblink2.0. It is featured with multi-instance, unconstrained, and untrimmed, which makes it more challenging and offers a closer correspondence to real-world demands;
• New Framework: We propose a one-stage multi-person eyeblink detection method InstBlink. It can jointly perform face detection, tracking, and instance-level eyeblink detection. Such a task-joint paradigm can benefit the sub-tasks uniformly. Benefited from the one-stage design, InstBlink also shows high efficiency especially in multi-instance scenarios.
• Open Source: We release the dataset, model, and code for training, testing, and demo, aiming to foster progress in relevant research fields.
• 🔥 Dataset Latest Update: We upgrade MPEblink 1.0 to MPEblink 2.0 by providing higher-quality annotations, expanding scenario diversity with a particular emphasis on dense and long-video settings, and introducing an enhanced evaluation metric, IB, as a unified metric that mitigates the bias of the two standalone metrics used in the conference version.
• 🔥 Model Latest Update: We propose InstBlink++ with disentangled fine-grained eye modeling and global–local feature interaction, achieving substantial performance gains while preserving the real-time advantages, which significant performance improvements on both instance localization and eyeblink detection.

1. Create a new conda environment:

   conda create -n instblink python=3.9
   conda activate instblink

2. Install Pytorch (1.7.1 is recommended), scipy, tqdm, pandas.

3. Install MMDetection.

   • Install MMCV first. 1.4.8 is recommended.

   • cd MPEblink
     pip install -v -e .

1. Download the MPEblink 1.0 & MPEblink2.0. Remember to change the dataset root path into yours in configs/base/mpeblink.py.

2. Convert the videos to raw frames.

   python tools/dataset_converters/mpeblink_build_raw_frames_dataset.py --root $YOUR_DATA_PATH

You can put some videos in demo_video/source_video/ and get the visualization inference result in demo_video/visual_result/ by running the following command:

bash tools/code_for_demo/demo.sh

• You can download the pre-trained model at Google Drive or Baidu Drive (code avk9) and put it in the pretrained_models directory.

• Run test.sh for inference and evaluation. Remember to change the dataset path into yours.

  bash tools/test_eval.sh

• Download the pretrained tevit_r50 model and place it in the pretrained_models directory.

• Run train.sh to begin training.

  bash tools/train.sh

• You can download the pre-trained model at quark Drive and put it in the pretrained_models directory.

• Run test_eval_instblink_plus.sh for inference and evaluation. Remember to change the dataset path into yours.

  bash tools/test_eval_instblink_plus.sh

• Download the pretrained tevit_r50 model and place it in the pretrained_models directory.

• Run train_instblink_plus.sh to begin training.

  bash tools/train_instblink_plus.sh

This code is inspired by TeViT and MMDetection. Thanks for their great contributions on the computer vision community.

If you find our work useful in your research, please consider to cite our paper:

@inproceedings{zeng2023real,
  title={Real-time Multi-person Eyeblink Detection in the Wild for Untrimmed Video},
  author={Zeng, Wenzheng and Xiao, Yang and Wei, Sicheng and Gan, Jinfang and Zhang, Xintao and Cao, Zhiguo and Fang, Zhiwen and Zhou, Joey Tianyi},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={13854--13863},
  year={2023}
}