Skip to main content
Springer Nature Link
Log in

Unsupervised anomaly detection and localization via bidirectional knowledge distillation

  • Original Article
  • Published:
Save article
View saved research
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Knowledge distillation has demonstrated significant potential in addressing the challenge of unsupervised anomaly detection (AD). The representation discrepancy of anomalies in the teacher–student (T-S) model provides evidence for anomaly detection and localization. However, the teacher model is pretrained for classification, while the anomaly scores in the distillation-based anomaly detection method are indirectly derived from the classification scores. The mismatch between the two tasks can hinder the optimization of the model. To tackle this issue, we propose an innovative bidirectional knowledge distillation model. In this approach, forward knowledge distillation is pivotal in bolstering the model’s capacity for generalization. Simultaneously, backward knowledge distillation promotes diversity in representing anomalies. This reciprocal knowledge exchange effectively wards off potential performance declines due to target inconsistency. Through bidirectional knowledge distillation, we establish a more encompassing and resilient framework for knowledge transfer. Additionally, we introduce a novel data augmentation strategy to simulate anomalies and effectively eliminate unnecessary noise. In experiments on the MVTec AD, the proposed model achieves competitive results compared to state-of-the-art methods, 97.47% on image-level AUC, 98.23% on pixel-level AUC, and 94.77% on instance-level PRO. These results demonstrate the practicality of our approach in anomaly detection and localization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from €37.37 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price includes VAT (Netherlands)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
The alternative text for this image may have been generated using AI.
Fig. 2
The alternative text for this image may have been generated using AI.
Fig. 3
The alternative text for this image may have been generated using AI.
Fig. 4
The alternative text for this image may have been generated using AI.
Fig. 5
The alternative text for this image may have been generated using AI.
Fig. 6
The alternative text for this image may have been generated using AI.
Fig. 7
The alternative text for this image may have been generated using AI.
Fig. 8
The alternative text for this image may have been generated using AI.
Fig. 9
The alternative text for this image may have been generated using AI.
Fig. 10
The alternative text for this image may have been generated using AI.

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Data availibility

Not applicable.

References

  1. Bergmann P, Fauser M, Sattlegger D, Steger C (2019) Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9592–9600

  2. Hou J, Zhang Y, Zhong Q, Xie D, Pu S, Zhou H (2021) Divide-and-assemble: Learning block-wise memory for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8791–8800

  3. Tao X, Zhang D, Ma W, Hou Z, Lu Z, Adak C (2022) Unsupervised anomaly detection for surface defects with dual-siamese network. IEEE Trans Industr Inf 18(11):7707–7717

    Article  Google Scholar 

  4. Zavrtanik V, Kristan M, Skočaj D (2021) Draem-a discriminatively trained reconstruction embedding for surface anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8330–8339

  5. Tang Y, Zhao L, Zhang S, Gong C, Li G, Yang J (2020) Integrating prediction and reconstruction for anomaly detection. Pattern Recogn Lett 129:123–130

    Article  Google Scholar 

  6. Cao Y, Wan Q, Shen W, Gao L (2022) Informative knowledge distillation for image anomaly segmentation. Knowl-Based Syst 248:108846

    Article  Google Scholar 

  7. Salehi M, Sadjadi N, Baselizadeh S, Rohban MH, Rabiee HR (2021) Multiresolution knowledge distillation for anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14902–14912

  8. Wang G, Han S, Ding E, Huang D (2021) Student-teacher feature pyramid matching for anomaly detection. arXiv preprint arXiv:2103.04257

  9. Deng H, Li X (2022) Self-supervised anomaly detection with random-shape pseudo-outliers. In: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 4768–4772 IEEE

  10. Li C-L, Sohn K, Yoon J, Pfister T (2021) Cutpaste: Self-supervised learning for anomaly detection and localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9664–9674

  11. Cao Y, Song Y, Xu X, Li S, Yu Y, Zhang Y, Shen W (2022) Semi-supervised knowledge distillation for tiny defect detection. In: 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 1010–1015. IEEE

  12. Bergmann P, Fauser M, Sattlegger D, Steger C (2020) Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4183–4192

  13. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. In: NIPS Deep Learning and Representation Learning Workshop. arxiv.org/abs/1503.02531

  14. Deng H, Li X (2022) Anomaly detection via reverse distillation from one-class embedding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9737–9746 (2022)

  15. Roth K, Pemula L, Zepeda J, Schölkopf B, Brox T, Gehler P (2022) Towards total recall in industrial anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14318–14328

  16. Heckler L, König R, Bergmann P (2023) Exploring the importance of pretrained feature extractors for unsupervised anomaly detection and localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2916–2925

  17. Pang G, Shen C, Cao L, Hengel AVD (2021) Deep learning for anomaly detection: a review. ACM Comput Surv (CSUR) 54(2):1–38

    Article  Google Scholar 

  18. Yun S, Han D, Oh SJ, Chun S, Choe J, Yoo Y (2019) Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023–6032

  19. DeVries T, Taylor GW (2017) Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552

  20. Zhong Z, Zheng L, Kang G, Li S, Yang Y (2020) Random erasing data augmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 13001–13008

  21. Perlin K (1985) An image synthesizer. ACM Siggraph Comput Gr 19(3):287–296

    Article  Google Scholar 

  22. Mishra P, Verk R, Fornasier D, Piciarelli C, Foresti GL (2021) Vt-adl: A vision transformer network for image anomaly detection and localization. In: 2021 IEEE 30th International Symposium on Industrial Electronics (ISIE), pp. 01–06. IEEE

  23. Abati D, Porrello A, Calderara S, Cucchiara R (2013) Latent space autoregression for novelty detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 481–490

  24. Kim KH, Shim S, Lim Y, Jeon J, Choi J, Kim B, Yoon AS (2020) Rapp: Novelty detection with reconstruction along projection pathway. In: International Conference on Learning Representations

  25. Akçay S, Atapour-Abarghouei A, Breckon TP (2019) Skip-ganomaly: Skip connected and adversarially trained encoder-decoder anomaly detection. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE

  26. Gong D, Liu L, Le V, Saha B, Mansour MR, Venkatesh S, Hengel Avd (2019) Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1705–1714

  27. Ristea N-C, Madan N, Ionescu RT, Nasrollahi K, Khan FS, Moeslund TB, Shah M (2022) Self-supervised predictive convolutional attentive block for anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13576–13586

  28. Liu W, Luo W, Lian D, Gao S (2018) Future frame prediction for anomaly detection–a new baseline. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6536–6545

  29. Wu P, Wang W, Chang F, Liu C, Wang B (2023) Dss-net: Dynamic self-supervised network for video anomaly detection. IEEE Trans Multimed

  30. Cao Y, Xu X, Liu Z, Shen W (2023) Collaborative discrepancy optimization for reliable image anomaly localization. IEEE Trans Ind Inf

  31. Golan I, El-Yaniv R (2018) Deep anomaly detection using geometric transformations. Adv Neural Inf Process Syst 31

  32. Yoo J, Zhao T, Akoglu L (2022) Self-supervision is not magic: Understanding data augmentation in image anomaly detection. https://api.semanticscholar.org/CorpusID:252918149

  33. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. Ieee

  34. Zeiler Matthew D, Dilip Krishnan GWT, Fergus R (2010) Deconvolutional networks. In: In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2528–2535

  35. Tung F, Mori G (2019) Similarity-preserving knowledge distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1365–1374

  36. Wan Q, Gao L, Li X, Wen L (2021) Industrial image anomaly localization based on Gaussian clustering of pretrained feature. IEEE Trans Industr Electron 69(6):6182–6192

    Article  Google Scholar 

  37. Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y (2014) Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550

  38. Loshchilov I, Hutter F (2018) Decoupled weight decay regularization. In: International Conference on Learning Representations

  39. Cohen N, Hoshen Y (2020) Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint arXiv:2005.02357

  40. Defard T, Setkov A, Loesch A, Audigier R (2021) Padim: a patch distribution modeling framework for anomaly detection and localization. In: Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10–15, 2021, Proceedings, Part IV, pp. 475–489. Springer

  41. Bae J, Lee J-H, Kim S (2023) Pni: industrial anomaly detection using position and neighborhood information. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6373–6383

  42. Liu Z, Zhou Y, Xu Y, Wang Z (2023) Simplenet: A simple network for image anomaly detection and localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20402–20411

  43. Yi J, Yoon S (2020) Patch svdd: Patch-level svdd for anomaly detection and segmentation. In: Proceedings of the Asian Conference on Computer Vision

  44. Lei J, Hu X, Wang Y, Liu D (2023) Pyramidflow: High-resolution defect contrastive localization using pyramid normalizing flow. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14143–14152

  45. Zhao M, Song Y (2023) Abnormal-aware loss and full distillation for unsupervised anomaly detection based on knowledge distillation. In: 2023 IEEE International Conference on Image Processing (ICIP), pp. 715–719. IEEE

Download references

Acknowledgements

This study was funded by Natural Science Foundation of Shanghai (22ZR1443700).

Author information

Authors and Affiliations

  1. School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, China

    Xiaoming Wang, Yongxiong Wang, Zhiqun Pan & Guangpeng Wang

Authors
  1. Xiaoming Wang
    View author publications

    Search author on:PubMed Google Scholar

  2. Yongxiong Wang
    View author publications

    Search author on:PubMed Google Scholar

  3. Zhiqun Pan
    View author publications

    Search author on:PubMed Google Scholar

  4. Guangpeng Wang
    View author publications

    Search author on:PubMed Google Scholar

Contributions

Xiaoming Wang and Yongxiong Wang were involved in conceptualization; Xiaoming Wang and Zhiqun Pan helped in methodology; Xiaoming Wang assisted in formal analysis and investigation; Xiaoming Wang helped in writing—original draft preparation; Yongxiong Wang, Zhiqun Pan, and Guangpeng Wang helped in writing—review and editing; funding acquisition was done by Yongxiong Wang. The manuscript is approved by all authors for publication

Corresponding author

Correspondence to Yongxiong Wang.

Ethics declarations

Conflict of interests

The authors have no relevant financial or nonfinancial interests to disclose.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, X., Wang, Y., Pan, Z. et al. Unsupervised anomaly detection and localization via bidirectional knowledge distillation. Neural Comput & Applic 36, 18499–18514 (2024). https://doi.org/10.1007/s00521-024-10172-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1007/s00521-024-10172-8

Keywords

Profiles

  1. Yongxiong Wang View author profile
  2. Guangpeng Wang View author profile