微信服务号

微信订阅号

2025年3月30日 21:02 星期日
首页 > 过刊浏览>2021年第32卷第11期 >3468-3481. DOI:10.13328/j.cnki.jos.006057
PDF HTML阅读 XML下载 导出引用 引用提醒
自动化张量分解加速卷积神经网络
作者:
作者简介:

宋冰冰(1994-),男,博士生,CCF学生会员,主要研究领域为深度学习,模型加速.
刘俊晖(1980-),男,博士,讲师,CCF专业会员,主要研究领域为模型驱动开发,深度学习,计算机视觉.
张浩(1992-),男,硕士,主要研究领域为深度学习,生物信息学.
梁宇(1964-),男,教授,主要研究领域为网络技术,虚拟化,云计算.
吴子锋(1996-),男,硕士生,主要研究领域为模型压缩与分解.
周维(1974-),男,博士,教授,博士生导师,CCF专业会员,主要研究领域为分布式处理.

通讯作者:

周维,E-mail:zwei@ynu.edu.cn

中图分类号:

TP183

基金项目:

国家自然科学基金（61762089，61863036，61663047）


Automated Tensor Decomposition to Accelerate Convolutional Neural Networks
Author:
Fund Project:

National Natural Science Foundation of China (61762089, 61863036, 61663047)

  • 摘要
    • | |
  • 访问统计
    • |
  • 参考文献 [48]
    • |
  • 相似文献 [20]
    • | | |
  • 文章评论
    摘要:

    近年来，卷积神经网络（CNN）展现了强大的性能，被广泛应用到了众多领域.由于CNN参数数量庞大，且存储和计算能力需求高，其难以部署在资源受限设备上.因此，对CNN的压缩和加速成为一个迫切需要解决的问题.随着自动化机器学习（AutoML）的研究与发展，AutoML对神经网络发展产生了深远的影响.受此启发，提出了基于参数估计和基于遗传算法的两种自动化加速卷积神经网络算法.该算法能够在给定精度损失范围内自动计算出最优的CNN加速模型，有效地解决了张量分解中，人工选择秩带来的误差问题，能够有效地提升CNN的压缩和加速效果.通过在MNIST和CIFAR-10数据集上的严格测试，与原网络相比，在MNIST数据集上准确率稍微下降了0.35%，模型的运行时间获得了4.1倍的大幅提升；在CIFAR-10数据集上，准确率稍微下降了5.13%，模型的运行时间获得了0.8倍的大幅提升.

    Abstract:

    Recently, convolutional neural network (CNN) have demonstrated strong performance and are widely used in many fields. Due to the large number of CNN parameters and high storage and computing power requirements, it is difficult to deploy on resource-constrained devices. Therefore, compression and acceleration of CNN models have become an urgent problem to be solved. With the research and development of automatic machine learning (AutoML), AutoML has profoundly impacted the development of neural networks. Inspired by this, this study proposes two automated accelerated CNN algorithms based on parameter estimation and genetic algorithms, which can calculate the optimal accelerated CNN model within a given accuracy loss range, effectively solving the error caused by artificially selected rank in tensor decomposition. It can effectively improve the compression and acceleration effects of the convolutional neural network. By rigorous testing on the MNIST and CIFAR-10 data sets, the accuracy rate on the MNIST dataset is slightly reduced by 0.35% compared to the original network, and the running time of the model is greatly reduced by 4.1 times, the accuracy rate on the CIFAR-10 dataset dropped slightly by 5.13%, and the running time of the model was greatly decreased by 0.8 times.

    参考文献
    [1] LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. IEEE Institute of Electrical and Electronics Engineers, 1998,86(11):2278-2324.
    [2] Bai C, Huang L, Chen JN, Pan X, Chen SY. Optimization of deep convolutional neural network for large scale image classification. Ruan Jian Xue Bao/Journal of Software, 2018,29(4):1029-1038(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5404.htm[doi:10.13328/j.cnki.jos.005404]
    [3] Zhou FY, Jin LP, Dong J. A review of convolutional neural networks. Chinese Journal of Computers, 2017,40(6):1229-1251(in Chinese with English abstract).[doi:10.11897/SP.J.1016.2017.01229]
    [4] Howard AG, Zhu M, Chen B, Kalenichenko D, Wang WJ, Weyand T, Andreetto M, Adam H. Mobilenets:Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
    [5] Ji RR, Lin SH, Chao F. A survey of deep neural network compression and acceleration. Journal of Computer Research and Development, 2018,55(9):1871-1888(in Chinese with English abstract).[doi:10.7544/issn1000-1239.2018.20180129]
    [6] Denil M, Shakibi B, Dinh L, Ranzato M, DeFreitas N. Predicting parameters in deep learning. In:Proc. of the 2013 MIT Press Conf. on Neural Information Processing Systems (NIPS). 2013. 2148-2156.
    [7] Welling M, Weber M. Positive tensor factorization. Pattern Recognition Letters, 2001,22(12):1255-1261.[doi:https://doi.org/10.1016/S0167-8655(01)00070-8]
    [8] Kim YD, Choi S. Nonnegative Tucker decomposition. In:Proc. of the 2007 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2007). 2007.
    [9] Hazan T, Polak S, Shashua A. Sparse image coding using a 3D non-negative tensor factorization. In:Proc. of the 2005 IEEE Conf. on 10th IEEE Int'l Conf. on Computer Vision (ICCV 2005). 2005. 50-57.[doi:10.1109/ICCV.2005.228]
    [10] Nion D, Sidiropoulos ND. Tensor algebra and multidimensional harmonic retrieval in signal processing for MIMO radar. IEEE Trans. on Signal Processing, 2010,58(11):5693-5705.
    [11] Benetos E, Kotropoulos C, Lidy T, Rauber A. Testing supervised classifiers based on non-negative matrix factorization to musical instrument classification. In:Proc. of the 14th European Signal Processing Conf. 2006. 1-5.
    [12] Chen Z, Lu Y. CubeSVD:A novel approach to personalized Web search*. In:Proc. of the 14th Int'l World Wide Web Conf. 2005. 382-390.[doi:10.1145/1060745.1060803]
    [13] Rendle S, Marinho LB, Nanopoulos A, Schmidt-Thieme L. Learning optimal ranking with tensor factorization for tag recommendation. In:Proc. of the KDD 2009. 2009. 727-736.
    [14] Xiong L, Chen X, Huang T, Schneider J, Carbonell J. Temporal collaborative filtering with Bayesian probabilistic tensor factorization. In:Proc. of the SIAM Int'l Conf. on Data Mining (SDM). 2010. 211-222.
    [15] Karatzoglou A, Amatriain X, Baltrunas L, Oliver N. Multiverse recommendation:N-dimensional tensor factorization for context-aware collaborative filtering. In:Proc. of the 4th ACM Conf. on Recommender Systems. ACM, 2010. 79-86.
    [16] Shi Y, Karatzoglou A, Baltrunas L, Larson M, Hanjalic A, Oliver N. TFMAP:Optimizing map for top-N context-aware recommendation. In:Proc. of the 35th Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval. 2012. 155-164.
    [17] LeCun Y, Denker JS, Solla S. Optimal Brain Damage. 1990. 598-605.
    [18] Hassibi B, Stork DG. Second order derivatives for network pruning:Optimal brain surgeon. In:Proc. of the 1993 MIT Press Conf. on Neural Information Processing Systems (NIPS). 1993. 164-171.
    [19] Srinivas S, Babu RV. Data-free parameter pruning for deep neural networks. arXiv preprint arXiv:1507.06149, 2015.
    [20] Han S, Mao H, Dally WJ. Deep compression:Compressing deep neural networks with pruning trained quantization and Huffman coding. arXiv preprint arXiv:1510.00149, 2015.
    [21] Han S, Pool J, Tran J, Dally W. Learning both weights and connections for efficient neural network. In:Proc. of the 2015 MIT Press Conf. on Neural Information Processing Systems (NIPS). Cambridge:MIT Press, 2015. 1135-1143.
    [22] Lebedev V, Lempitsky V. Fast ConvNets using group-wise brain damage. In:Proc. of the 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2016). Piscataway:IEEE, 2016. 2554-2564.
    [23] Molchanov P, Tyree S, Karras T, Aila T, Kautz J. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2017.
    [24] Gong Y, Liu L, Yang M, Bourdev L. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.
    [25] Gupta S, Agrawal A, Gopalakrishnan K, Narayanan P. Deep learning with limited numerical precision. In:Proc. of the 32nd Int'l Conf. on Machine Learning (ICML). 2015. 1737-1746.
    [26] Li F, Zhang B, Liu B. Ternary weight networks. arXiv preprint arXiv:1605.04711, 2016.
    [27] Courbariaux M, Bengio Y, David J. BinaryConnect:Training deep neural networks with binary weights during propagations. In:Proc. of the 2015 MIT Press Conf. on Neural Information Processing Systems (NIPS). 2015. 3123-3131.
    [28] Chollet F. Xception:Deep learning with depthwise separable convolutions. In:Proc. of the 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2017. 1800-1807.
    [29] Zhang X, Zhou X, Lin M, Sun J. ShuffleNet:An extremely efficient convolutional neural network for mobile devices. In:Proc. of the 2018 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2018. 6848-6856.
    [30] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
    [31] Roberto R, Sironi A, Lepetit V, Fua P. Learning separable filters. In:Proc. of the 2013 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2013. 2754-2761.
    [32] Denton E, Zaremba W, Bruna J. Exploiting linear structure within convolutional networks for efficient evaluation. In:Proc. of the 2014 MIT Press Conf. on Neural Information Processing Systems (NIPS). 2014. 1269-1277.
    [33] Jaderberg M, Vedaldi A, Zisserman A. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.
    [34] Lebedev V, Ganin Y, Rakhuba M, Oseledets I, Lempitsky V. Speeding-up convolutional neural networks using fine-tuned CP-decomposition. arXiv preprint arXiv:1412.6553, 2014.
    [35] Kim YD, Park E, Yoo S, Choi T, Yang L, Shin DJ. Compression of deep convolutional neural networks for fast and low power mobile applications. Computer Science, 2015,71(2):576-584.
    [36] Girshick R. Fast R-CNN. In:Proc. of the IEEE Int'l Conf. on Computer Vision (ICCV). 2015. 1440-1448.
    [37] Tai C, Xiao T, Zhang Y, Wang XG, E WN. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:1511.06067, 2016.
    [38] Wen W, Xu C, Wu C, Wang YD, Chen YR, Li H. Coordinating filters for faster deep neural networks. In:Proc. of the 2017 IEEE Int'l Conf. on Computer Vision (ICCV). 2017. 658-666.
    [39] Yao Q, Wang MS, Chen YQ, Dai WY, Li YF, Tu WW, Qiang Y, Yu Y. Taking human out of learning applications:A survey on automated machine learning. arXiv preprint arXiv:1810.13306, 2018.
    [40] Zoph B, Le QV. Neural architecture search with reinforcement learning. In:Proc. of the 5th Int'l Conf. on Learning Representations (ICLR). 2017.
    [41] Zhang HL, Kiranyaz S, Gabbouj M. Finding better topologies for deep convolutional neural networks by evolution. arXiv preprint arXiv:1809.03242v1, 2018.
    [42] Ma NN, Zhang XY, Zheng HT, Sun J. ShuffleNet V2:Practical guidelines for efficient CNN architecture design. In:Proc. of the European Conf. on Computer Vision (ECCV). 2018. 122-138.
    [43] He YH, Lin J, Liu ZJ, Wang HR, Li LJ, Han S. AMC:AutoML for model compression and acceleration on mobile devices. In:Proc. of the European Conf. on Computer Vision (ECCV). 2018. 815-832.
    [44] Liu ZC, Mu HY, Zhang XY, Guo ZC, Yang X, Cheng K, Sun J. MetaPruning:Meta learning for automatic neural network channel pruning. In:Proc of the International Conference on Computer Vision ICCV. 2019. 3295-3304.
    附中文参考文献:
    [2] 白琮,黄玲,陈佳楠,潘翔,陈胜勇.面向大规模图像分类的深度卷积神经网络优化.软件学报,2018,29(4):1029-1038. http://www. jos.org.cn/1000-9825/5404.htm[doi:10.13328/j.cnki.jos.005404]
    [3] 周飞燕,金林鹏,董军.卷积神经网络研究综述.计算机学报,2017,40(6):1229-1251.[doi:10.11897/SP.J.1016.2017.01229]
    [5] 纪荣嵘,林绍辉,晁飞.深度神经网络压缩与加速综述.计算机研究与发展,2018,55(9):1871-1888.[doi:10.7544/issn1000-1239. 2018.20180129]
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

宋冰冰,张浩,吴子锋,刘俊晖,梁宇,周维.自动化张量分解加速卷积神经网络.软件学报,2021,32(11):3468-3481

复制
分享
文章指标
  • 点击次数:752
  • 下载次数: 3458
  • HTML阅读次数: 1603
  • 引用次数: 0
历史
  • 收稿日期:2019-11-01
  • 最后修改日期:2020-02-05
  • 在线发布日期: 2021-11-05
  • 出版日期: 2021-11-06
文章二维码
友情链接:中国科学院软件研究所中国计算机学会中国科学院国家自然科学基金委员会CCF数字图书馆Int'l J of Softw Info计算机系统应用
您是第19754439位访问者
版权所有：中国科学院软件研究所 京ICP备05046678号-3
地址：北京市海淀区中关村南四街4号,邮政编码：100190
电话：010-62562563 传真：010-62562533 Email：jos@iscas.ac.cn
技术支持：北京勤云科技发展有限公司

京公网安备 11040202500063号