避免近期偏好的自学习掩码分区增量学习
作者:
作者单位:

1.西安工业大学;2.空军工程大学


Avoid Recent Bias Self-Learning Mask Partitioned Incremental Learning
Author:
Affiliation:

Xi’an Technological University

  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [50]
  • | |
  • 引证文献
  • | |
  • 文章评论
    摘要:

    忘是人工神经网络在增量学习中的最大问题,被称为“灾难性遗忘”。而人类可以持续地获取新知识,对旧知识基本不遗忘。人类的这种能持续“增量学习”而很少遗忘是与人脑具有分区学习结构和记忆回放能力相关的。为模拟人脑的这种结构和能力,本文提出一种“避免近期偏好的自学习掩码分区增量学习方法”简称ASPIL。它包含“区域隔离”和“区域集成”两阶段,二者交替迭代实现持续的增量学习。首先,本文提出“BN稀疏区域隔离”方法,将新的学习过程与现有知识隔离,避免干扰现有知识;对于“区域集成”,提出自学习掩码(SLM)和双分支融合(GBF)方法。其中SLM准确提取新知识,并提高网络对新知识的适应性,而 GBF将新旧知识融合,以达到建立统一的、高精度的认知的目的;训练时,为确保进一步兼顾旧知识,避免对新知识的偏好,提出边缘损失正则项来避免“近期偏好”问题。为评估以上所提出方法的效用,在增量学习标准数据集CIFAR-100和miniImageNet上系统地进行了消融实验,并与最新的一系列知名方法进行了比较。实验结果表明,本文方法提高了人工神经网络的记忆能力,与最新知名方法相比识别率平均提升了5.27%以上。

    Abstract:

    Forgetting is the biggest problem of artificial neural network in incremental learning, which is called "disaster forgetting", while humans can continuously acquire new knowledge without forgetting old knowledge. This continuous "incremental learning" of human without forgetting is related to the human brain's partitioned learning structure and memory playback ability. To simulate the human brain's partition learning and memory replay, this paper proposes an incremental learning approach of "Avoid Recent Biase Self-Learning Mask Partitioned Incremental Learning ", referred to as ASPIL. ASPIL is a two-stage strategy that comprises alternate iterations of “regional isolation” and “regional integration” in order to accomplish continuous class incremental learning. This paper proposes the "BN sparse region isolation" method, which isolates the new learning process from the existing knowledge and avoids interfering with the existing knowledge; For "regional ensemble", self-learning mask (SLM) and dual-branch fusion (GBF) methods are proposed. SLM accurately extracts new knowledge and improves the adaptability of the network to new knowledge, while GBF integrates the old and new knowledge to achieve the goal of establishing a unified and high-precision cognition; During training, in order to further take into account, the old knowledge and avoid the preference for new knowledge, an edge loss regular term is proposed to avoid the "recent bias" problem. To evaluate the positive utility of the proposed methods above, ablation experiments are systematically performed on the incremental learning standard datasets CIFAR-100 and miniImageNet, and compared with a series of state-of-the-art methods. The experimental results show that the method in this paper improves the memory ability of the artificial neural network, and the recognition rate is increased by more than 5.27% on average compared with the latest well-known methods. The experimental results show that this method improves the memory ability of the artificial neural network, and the recognition rate is increased by more than 5.27% on average compared with the latest well-known methods.

    参考文献
    [1] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019, 113: 54-71.
    [2] Randall C, O’Reilly, Rajan Bhattacharyya, Michael D, Howard, and Nicholas Ketza. Complementary learning systems. Cognitive science, 2014, 38(6): 1229-1248.
    [3] Randall C, O’Reilly, Rajan Bhattacharyya, Michael D, Howard, and Nicholas Ketza. Complementary learning systems. Cognitive science, 2014, 38(6): 1229-1248.
    [4] Joseph O’Neill, Barty Pleydell-Bouverie, David Dupret and Jozsef Csicsvari. Play it again: reactivation of waking experience and memory. Trends in neurosciences, 2010, 33(5): 220-229.
    [5] Gail A Carpenter and Stephen Grossberg. The ART of adaptive pattern recognition by a self-organizing neural network. Computer, 1988, 21(3): 77-88.
    [6] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 1989, 24: 109-165.
    [7] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1999, 3(4): 128-135.
    [8] Robi Polikar, Lalita Udpa, Satish S Udpa, and Vasant Honavar. Learn++: An incremental learning algorithm for supervised neural networks. IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews), 2001, 31(4): 497-508.
    [9] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017, 114(13): 3521–3526.
    [10] Zifeng Wang, Tong Jian, Kaushik Chowdhury, Yanzhi Wang, Jennifer Dy, and Stratis Ioannidis. Learn-prune-share for lifelong learning//2020 IEEE International Conference on Data Mining (ICDM). Sorrento, Italy, 2020: 641-650.
    [11] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning//In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA, 2021: 2544-2553.
    [12] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing//In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA, 2019: 831-839.
    [13] Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning//In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA, 2021: 3589-3599.
    [14] James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: A new approach for data-free class-incremental learning//In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada, 2021: 9374-9384.
    [15] Francisco M Castro, Manuel J Marin-Jimenez, Nicolas Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning//In Proceedings of the European conference on computer vision (ECCV). Munich, Germany, 2018: 233-248.
    [16] Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Lifelong generative modeling. Neurocomputing, 2020, 404: 381-400.
    [17] Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting//2018 24th International Conference on Pattern Recognition (ICPR). IEEE. Beijing, China, 2018: 2262-2268.
    [18] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning//In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA, 2019: 374-382.
    [19] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 2009, 1(4).
    [20] Oriol Vinyals, Charles Blundell, and Timothy Lillicrap. Matching networks for one shot learning. Advances in neural information processing systems, 2016, 29.
    [21] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(12): 2935-2947.
    [22] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, and Raia Hadsel. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 2017, 114(13): 3521-3526.
    [23] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531, 2015.
    [24] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018.
    [25] Amal Rannen, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning//In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Venice, Italy, 2017: 1320-1328.
    [26] Jonathan Schwarz, Jelena Luketina, Wojciech M Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning//International Conference on Machine Learning. Stockholm, Sweden, 2018: 4528-4537.
    [27] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence//International Conference on Machine Learning. Sydney, Australia, 2017: 3987-3995.
    [28] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory Aware Synapses: Learning what (not) to forget//In Proceedings of the European conference on computer vision (ECCV). Munich, Germany, 2018: 139-154.
    [29] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, Rama Chellappa. Learning without memorizing//In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA, 2019: 5138-5146.
    [30] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C.-C. Jay Kuo. Class-incremental learning via deep model consolidation//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Snowmass, USA, 2020: 1131-1140.
    [31] Steven C Y Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems, 2019, 32.
    [32] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.4671, 2016.
    [33] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017.
    [34] Nikhil Mehta, Kevin J Liang, and Lawrence Carin. Bayesian nonparametric weight factorization for continual learning. arXiv preprint arXiv:2004.10098, 2020.
    [35] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming//In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Venice, Italy, 2017: 2736-2744.
    [36] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Gcnet: Non-local networks meet squeeze-excitation networks and beyond//In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). Seoul, Korea, 2019: 1971-1980.
    [37] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks//In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA, 2018: 7794-7803.
    [38] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks//In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA, 2018: 7132-7141.
    [39] Jinting Chen, Zhaocheng Zhu, Cheng Li, and Yuming Zhao. Self-adaptive network pruning//In International Conference on Neural Information Processing(ICONIP). Sydney, Australia, 2019: 175-186.
    [40] Xiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong, Shaohua Tan, and Kuiyuan Yang. Gated fully fusion for semantic segmentation//Proceedings of the AAAI conference on artificial intelligence. New York, USA, 2020, 34(7): 11418-11425.
    [41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition//In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA, 2016: 770-778.
    [42] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
    [43] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. Icarl: Incremental classifier and representation learning//In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA, 2017: 2001-2010.
    [44] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip H S Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019.
    [45] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM//International Conference on Learning Representations (ICLR). New Orleans City, USA, 2019.
    [46] Miao Yongbiao. Research on Image Incremental Learning Based on Deep Learning. Zhejiang University of Technology, Zhe jiang, 2020 (in chinese)(缪永彪. 基于深度学习的图像增量学习研究[硕士学位论文]. 浙江工业大学, 浙江, 2020.)
    [47] He Li,Han Keping,Zhu Hongxi,and Liu Ying. Deep incremental image classification method based on double-branch iteration. Pattern Recognition and Artificial Intelligence, 2020, 33(2): 150-159 (in chinese)(何丽, 韩克平, 朱泓西, 刘颖. 双分支迭代的深度增量图像分类方法[J]. 模式识别与人工智能, 2020, 33(2): 150-159.)
    [48] Ding Siyu. Research on augmented class learning. Southeast University. Nan Jing, 2019 (in chinese)(丁思宇. 增量类学习若干问题研究[硕士学位论文]. 东南大学, 南京, 2019.)
    [49] Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 1995, 7(2): 123-146.
    [50] Han Ya′nan, Liu Jianwei, and Luo Xionglin. Research progress of continual learning. Journal of Computer Research and Development (in chinese)(韩亚楠, Jianwei Liu, Xiong-Lin Luo. 连续学习研究进展[J/OL]. 计算机研究与发展, available: http://kns.cnki.net/kcms/detail/11.1777.TP.20210824.1540.012.html 2022, 5, 3.)
    相似文献
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文
相关视频

分享
文章指标
  • 点击次数:67
  • 下载次数: 0
  • HTML阅读次数: 0
  • 引用次数: 0
历史
  • 收稿日期:2022-08-31
  • 最后修改日期:2023-02-13
  • 录用日期:2023-04-11
文章二维码
您是第位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号