Feature Constrained Restricted Distillation Learning for Visual Anomaly Detection
Author:
Affiliation:

  • Article
  • | |
  • Metrics
  • |
  • Reference [49]
  • |
  • Related [20]
  • | | |
  • Comments
    Abstract:

    This study proposes a new feature constrained distillation learning method for visual anomaly detection, which makes full use of the features of the teacher model to instruct the student model to efficiently identify abnormal images. Specifically, the vision transformer (ViT) model is introduced as the backbone network of anomaly detection tasks, and a central feature strategy is put forward to constrain the output features of the student network. Considering the strong feature expressiveness of the teacher network, the central feature strategy is developed to dynamically generate the feature representation centers of normal samples for the student network from the teacher network. In this way, the ability of the student network to describe the feature output of normal data is improved, and the feature difference between the student and teacher networks in abnormal data is widened. In addition, to minimize the difference between the student and teacher networks in the feature representation of normal images, the proposed method leverages the Gram loss function to constrain the relationship between the coding layers of the student network. Experiments are conducted on three general anomaly detection data sets and one real-world industrial anomaly detection data set, and the experimental results demonstrate that the proposed method significantly improves the performance of visual anomaly detection compared with the state-of-the-art methods.

    Reference
    [1] Domingues R, Filippone M, Michiardi P, Zouaoui J. A comparative evaluation of outlier detection algorithms: Experiments and analyses. Pattern Recognition, 2018, 74: 406-421. [doi: 10.1016/j.patcog.2017.09.037]
    [2] Pang GS, Shen CH, Cao LB, van den Hengel A. Deep learning for anomaly detection: A review. ACM Computing Surveys, 2022, 54(2): 38. [doi: 10.1145/3439950]
    [3] Ruff L, Vandermeulen R, Goernitz N, Deecke L, Siddiqui SA, Binder A, Müller E, Kloft M. Deep one-class classification. In: Proc. of the 35th Int’l Conf. on Machine Learning. Stockholm: PMLR, 2018. 4393-4402.
    [4] 宋杰, 肖亮, 练智超, 蔡子贇, 蒋国平. 基于深度学习的数字病理图像分割综述与展望. 软件学报, 2021, 32(5): 1427-1460. http://www.jos.org.cn/1000-9825/6205.htm
    Song J, Xiao L, Lian ZC, Cai ZY, Jiang GP. Overview and prospect of deep learning for image segmentation in digital pathology. Ruan Jian Xue Bao/Journal of Software, 2021, 32(5): 1427-1460 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6205.htm
    [5] 丁小欧, 于晟健, 王沐贤, 王宏志, 高宏, 杨东华. 基于相关性分析的工业时序数据异常检测. 软件学报, 2020, 31(3): 726-747. http://www.jos.org.cn/1000-9825/5907.htm
    Ding XO, Yu SJ, Wang MX, Wang HZ, Gao H, Yang DH. Anomaly detection on industrial time series based on correlation analysis. Ruan Jian Xue Bao/Journal of Software, 2020, 31(3): 726-747 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5907.htm
    [6] An J, Cho S. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, 2015, 2(1): 1-18. (查阅所有网上资料, 未能确认本条文献修改是否正确, 请联系作者确认)
    [7] Zhou C, Paffenroth RC. Anomaly detection with robust deep autoencoders. In: Proc. of the 23rd ACM SIGKDD Int’l Conf. on Knowledge Discovery and Data Mining. Halifax: ACM, 2017. 665-674.
    [8] Bergmann P, Fauser M, Sattlegger D, Steger C. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In: Proc. of the 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020. 4182-4191.
    [9] Salehi M, Sadjadi N, Baselizadeh S, Rohban MH, Rabiee HR. Multiresolution knowledge distillation for anomaly detection. In: Proc. of the 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021. 14897-14907.
    [10] Yim J, Joo D, Bae J, Kim J. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In: Proc. of the 2017 IEEE Conf. on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017. 7130-7138.
    [11] Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. In: Proc. of the 27th Int’l Conf. on Neural Information Processing Systems. Montreal: MIT Press, 2014. 2672-2680.
    [12] Zhao H, Gallo O, Frosio I, Kautz J. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging, 2017, 3(1): 47-57. [doi: 10.1109/TCI.2016.2644865]
    [13] Liu W, Luo WX, Lian DZ, Gao SH. Future frame prediction for anomaly detection—A new baseline. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 6536-6545.
    [14] Gong D, Liu LQ, Le V, Saha B, Mansour RM, Venkatesh S, van den Hengel A. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In: Proc. of the 2019 IEEE/CVF Int’l Conf. on Computer Vision. Seoul: IEEE, 2019. 1705-1714.
    [15] Park H, Noh J, Ham B. Learning memory-guided normality for anomaly detection. In: Proc. of the 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020. 14360-14369.
    [16] Ye F, Huang CQ, Cao JK, Li MS, Zhang Y, Lu CW. Attribute restoration framework for anomaly detection. IEEE Transactions on Multimedia, 2020, 24: 116-127. [doi: 10.1109/TMM.2020.3046884]
    [17] Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. In: Proc. of the 18th Int’l Conf. on Medical Image Computing and Computer-assisted Intervention. Munich: Springer, 2015. 234-241.
    [18] Horé A, Ziou D. Image quality metrics: PSNR vs. SSIM. In: Proc. of the 20th Int’l Conf. on Pattern Recognition. Istanbul: IEEE, 2010. 2366-2369. [doi:DOI: 10.1109/ICPR.2010.579]
    [19] Schlegl T, Seeböck P, Waldstein SM, Schmidt-Erfurth U, Langs G. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: Proc. of the 25th Int’l Conf. on Information Processing in Medical Imaging. Boone: Springer, 2017. 146-157.
    [20] Perera P, Nallapati R, Xiang B. OCGAN: One-class novelty detection using GANs with constrained latent representations. In: Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 2893-2901.
    [21] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.
    [22] Sabokrou M, Fayyaz M, Fathy M, Moayed Z, Klette R. Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes. Computer Vision and Image Understanding, 2018, 172: 88-97. [doi: 10.1016/j.cviu.2018.02.006]
    [23] DosoViTskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai XH, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N. An image is worth 16x16 words: Transformers for image recognition at scale. In: Prof. of the 9th Int’l Conf. on Learning Representations. OpenReview.net, 2021. 1-22.
    [24] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez NA, Kaiser L, Polosukhin I. Attention is all you need. In: Proc. of the 31st Int’l Conf. on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017. 6000-6010.
    [25] Neimark D, Bar O, Zohar M, Asselmann D. Video transformer network. In: Proc. of the 2021 IEEE/CVF Int’l Conf. on Computer Vision Workshops. Montreal: IEEE, 2021. 3156-3165.
    [26] Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S. End-to-end object detection with transformers. In: Proc. of the 16th European Conf. on Computer Vision. Glasgow: Springer, 2020. 213-229.
    [27] Beal J, Kim E, Tzeng E, Park DH, Zhai A, Kislyuk D. Toward transformer-based object detection. arXiv:2012.09958, 2020.
    [28] Hu RH, Singh A. UniT: Multimodal multitask learning with a unified transformer. arXiv:2102.10772, 2021.
    [29] Deng J, Dong W, Socher R, Li LJ, Li K, Li FF. ImageNet: A large-scale hierarchical image database. In: Proc. of the 2009 IEEE Conf. on Computer Vision and Pattern Recognition. Miami: IEEE, 2009. 248-255.
    [30] Krizhevsky A. Learning multiple layers of features from tiny images. Technical Report, Toronto: University of Toronto, 2009.
    [31] Xiao H, Rasul K, Vollgraf R. Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747, 2017.
    [32] Bergmann P, Fauser M, Sattlegger D, Steger C. MVTec AD—A comprehensive real-world dataset for unsupervised anomaly detection. In: Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 9584-9592.
    [33] Chen YQ, Zhou XS, Huang TS. One-class SVM for learning in image retrieval. In: Proc of the 2001 Int’l Conf. on Image Processing. Thessaloniki: IEEE, 2001. 34-37.
    [34] Li XY, Kiringa I, Yeap T, Zhu XD, Li YF. Exploring deep anomaly detection methods based on capsule net. In: Proc. of the 33rd Canadian Conf. on Artificial Intelligence. Ottawa: Springer, 2020. 375-387.
    [35] Abati D, Porrello A, Calderara S, Cucchiara R. Latent space autoregression for novelty detection. In: Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 481-490.
    [36] Goyal S, Raghunathan A, Jain M, Simhadri HV, Jain P. DROCC: Deep robust one-class classification. In: Proc. of the 37th Int’l Conf. on Machine Learning. Vienna: PMLR, 2020. 3711-3721.
    [37] Golan I, El-Yaniv R. Deep anomaly detection using geometric transformations. In: Proc. of the 32nd Int’l Conf. on Neural Information Processing Systems. Montréal: Curran Associates Inc., 2018. 9781-9791.
    [38] Zhai SF, Cheng Y, Lu WN, Zhang ZF. Deep structured energy based models for anomaly detection. In: Prof. of the 33rd Int’l Conf. on Machine Learning. New York City: JMLR.org, 2016. 1100-1109.
    [39] Zong B, Song Q, Min MR, Cheng W, Lumezanu C, Cho DK, Chen HF. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: Prof. of the 6th Int’l Conf. on Learning Representations. Vancouver: OpenReview.net, 2018. 1-19.
    [40] Sabokrou M, Khalooei M, Fathy M, Adeli E. Adversarially learned one-class classifier for novelty detection. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 3379-3388.
    [41] Deecke L, Vandermeulen R, Ruff L, Mandt S, Kloft M. Image anomaly detection with generative adversarial networks. In: Proc. of the 2018 European Conf. on Machine Learning and Knowledge Discovery in Databases. Dublin: Springer, 2018. 3-17.
    [42] Akcay S, Atapour-Abarghouei A, Breckon TP. GANomaly: Semi-supervised anomaly detection via adversarial training. In: Proc. of the 14th Asian Conf. on Computer Vision. Perth: Springer, 2018. 622-637.
    [43] Sabokrou M, Pourreza M, Fayyaz M, Entezari R, Fathy M, Gall J, Adeli E. AVID: Adversarial visual irregularity detection. In: Proc. of the 14th Asian Conf. on Computer Vision. Perth: Springer, 2018. 488-505.
    [44] Bergmann P, Löwe S, Fauser M, Sattlegger D, Steger C. Improving unsupervised defect segmentation by applying structural similarity to autoencoders. In: Prof. of the 14th Int’l Joint Conf. on Computer Vision, Imaging and Computer Graphics Theory and Applications. Prague: SciTePress, 2019. 372-380.
    [45] Dehaene D, Frigo O, Combrexelle S, Eline P. Iterative energy-based projection on a normal data manifold for anomaly localization. In: Proc. of the 8th Int’l Conf. on Learning Representations. Addis Ababa: OpenReview.net, 2020. 1-17.
    [46] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. In: Proc. of the 3rd Int’l Conf. on Learning Representations. San Diego: ICLR, 2015. 1-14.
    [47] He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. In: Proc. of the 2016 Conf. on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016. 770-778.
    Cited by
    Comments
    Comments
    分享到微博
    Submit
Get Citation

邢鹏,蒋鑫,潘永华,唐金辉,李泽超.基于特征约束蒸馏学习的视觉异常检测.软件学报,2023,34(9):4378-4391

Copy
Share
Article Metrics
  • Abstract:891
  • PDF: 2840
  • HTML: 1130
  • Cited by: 0
History
  • Received:June 26,2021
  • Revised:September 27,2021
  • Online: December 22,2022
  • Published: September 06,2023
You are the first2044074Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address:4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code:100190
Phone:010-62562563 Fax:010-62562533 Email:jos@iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063