DeepRanger: Coverage-guided Deep Forest Testing Approach
Author:
Affiliation:

Clc Number:

TP311

  • Article
  • | |
  • Metrics
  • |
  • Reference [33]
  • |
  • Related
  • | | |
  • Comments
    Abstract:

    Comparing with traditional software, the deep learning software has different structures. Even if a lot of test data is used for testing the deep learning software, the adequacy of testing still hard to be evaluted, and many unknown defects could be implied. The deep forest is an emerging deep learning model that overcomes many shortcomings of deep neural networks. For example, the deep neural network requires a lot of training data, high performance computing platform, and many hyperparameters. However, there is no research on testing deep forest. Based on the structural characteristics of deep forests, this study proposes a set of testing coverage criteria, including random forest node coverage (RFNC), random forest leaf coverage (RFLC), cascad forest class coverage (CFCC), and cascad forest output coverage (CFOC). DeepRanger, a coverage-oriented test data generation method based on genetic algorithm, is proposed to automatically generate new test data and effectively improve the model coverage of the test data. Experiments are carried out on the MNIST data set and the gcForest, which is an open source deep forest project. The experimental results show that the four coverage criteria proposed can effectively evaluate the adequacy of the test data set for the deep forest model. In addition, comparing with the genetic algorithm based on random selection, DeepRanger, which is guided by coverage information, can improve the testing coverage of the deep forest model under testing.

    Reference
    [1] 余凯, 贾磊, 陈雨强, 徐伟. 深度学习的昨天、今天和明天. 计算机研究与发展, 2013, 50(9): 1799–1804.
    Yu K, Jia L, Chen YQ, Xu W. Deep learning: Yesterday, today, and tomorrow. Journal of Computer Research and Development, 2013, 50(9): 1799–1804 (in Chinese with English abstract).
    [2] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436–444. [doi: 10.1038/nature14539]
    [3] Du XN, Xie XF, Li Y, Ma L, Liu Y, Zhao JJ. DeepStellar: Model-based quantitative analysis of stateful deep learning systems. In: Proc. of the 27th ACM Joint Meeting on European Software Engineering Conf. and Symp. on the Foundations of Software Engineering. Tallinn: ACM, 2019. 477–487.
    [4] Ziegler C. A Google self-driving car caused a crash for the first time: A bad assumption led to a minor fender-bender. 2016. http://www.theverge.com/2016/2/29/11134344/google-selfdriving-car-crash-report
    [5] BBC NEWS. Tesla autopilot crash driver “Was Playing Video Game”. 2020. https://www.bbc.com/news/technology-51645566
    [6] Ma L, Juefei-Xu F, Zhang FY, Sun JY, Xue MH, Li B, Chen CY, Su T, Li L, Liu Y, Zhao JJ, Wang YD. DeepGauge: Multi-granularity testing criteria for deep learning systems. In: Proc. of the 33rd ACM/IEEE Int’l Conf. on Automated Software Engineering. Montpellier: ACM, 2018. 120–131.
    [7] Xie XF, Ma L, Juefei-Xu F, Xue MH, Chen HX, Liu Y, Zhao JJ, Li B, Yin JX, See S. DeepHunter: A coverage-guided fuzz testing framework for deep neural networks. In: Proc. of the 28th ACM SIGSOFT Int’l Symp. on Software Testing and Analysis. Beijing: ACM, 2019. 146–157.
    [8] Zhou ZH, Feng J. Deep forest: Towards an alternative to deep neural networks. In: Proc. of the 26th Int’l Joint Conf. on Artificial Intelligence. Melbourne: IJCAI, 2017. 3553–3559.
    [9] Zhou ZH, Feng J. Deep forest. National Science Review, 2019, 6(1): 74–86. [doi: 10.1093/nsr/nwy108]
    [10] Zhu H, Hall PAV, May JHR. Software unit test coverage and adequacy. ACM Computing Surveys, 1997, 29(4): 366–427. [doi: 10.1145/267580.267590]
    [11] Holland JH. Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control and Artificial Intelligence. Cambridge: MIT Press, 1992.
    [12] LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278–2324. [doi: 10.1109/5.726791]
    [13] 朱锐, 王怀民, 冯大为. 基于偏好推荐的可信服务选择. 软件学报, 2011, 22(5): 852–864. http://www.jos.org.cn/1000-9825/3804.htm
    Zhu R, Wang HM, Feng DW. Trustworthy services selection based on preference recommendation. Ruan Jian Xue Bao/Journal of Software, 2011, 22(5): 852–864 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/3804.htm
    [14] Witten IH, Frank E. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. San Francisco: Morgan Kaufmann Publishers Inc., 2000.
    [15] 王赞, 闫明, 刘爽, 陈俊洁, 张栋迪, 吴卓, 陈翔. 深度神经网络测试研究综述. 软件学报, 2020, 31(5): 1255–1275. http://www.jos.org.cn/1000-9825/5951.htm
    Wang Z, Yan M, Liu S, Chen JJ, Zhang DD, Wu Z, Chen X. Survey on testing of deep neural networks. Ruan Jian Xue Bao/Journal of Software, 2020, 31(5): 1255–1275 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5951.htm
    [16] Goodfellow I, Papernot N. The challenge of verification and testing of machine learning. 2017. http://www.cleverhans.io/security/privacy/ml/2017/06/14/verification.html
    [17] Tian YC, Pei KX, Jana S, Ray B. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In: Proc. of the 40th Int’l Conf. on Software Engineering. Gothenburg: ACM, 2018. 303–314.
    [18] Zhang MS, Zhang YQ, Zhang LM, Liu C, Khurshid S. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In: Proc. of the 33rd IEEE/ACM Int’l Conf. on Automated Software Engineering. Montpellier: IEEE, 2018. 132–142.
    [19] Wang JY, Dong GL, Sun J, Wang XY, Zhang PX. Adversarial sample detection for deep neural network through model mutation testing. In: Proc. of the 41st IEEE/ACM Int’l Conf. on Software Engineering. Montreal: IEEE, 2019. 1245–1256.
    [20] Hayhurst KJ. A Practical Tutorial on Modified Condition/Decision Coverage. DIANE Publishing, 2001.
    [21] Sun YC, Huang XW, Kroening D, Sharp J, Hill M, Ashmore R. Testing deep neural networks. arXiv:1803.04792, 2018.
    [22] Pei KX, Cao YZ, Yang JF, Jana S. DeepXplore: Automated whitebox testing of deep learning systems. In: Proc. of the 26th Symp. on Operating Systems Principles. Shanghai: ACM, 2017. 1–18.
    [23] Ma L, Juefei-Xu F, Xue MH, Li B, Li L, Liu Y, Zhao JJ. DeepCT: Tomographic combinatorial testing for deep learning systems. In: Proc. of the 26th IEEE Int’l Conf. on Software Analysis, Evolution and Reengineering. Hangzhou: IEEE, 2019. 614–618.
    [24] Goodfellow IJ, Shlens J, Szegedy C. Explaining and harnessing adversarial examples. In: Proc. of the 3rd Int’l Conf. on Learning Representations. San Diego, 2015.
    [25] Kim J, Feldt R, Yoo S. Guiding deep learning system testing using surprise adequacy. In: Proc. of the 41st IEEE/ACM Int’l Conf. on Software Engineering. Montreal: IEEE, 2019. 1039–1049.
    [26] Li ZN, Ma XX, Xu C, Cao C. Structural coverage criteria for neural networks could be misleading. In: Proc. of the 41st IEEE/ACM Int’l Conf. on Software Engineering: New Ideas and Emerging Results. Montreal: IEEE, 2019. 89–92.
    [27] 卢喜东, 段哲民, 钱叶魁, 周巍. 一种基于深度森林的恶意代码分类方法. 软件学报, 2020, 31(5): 1454–1464. http://www.jos.org.cn/1000-9825/5660.htm
    Lu XD, Duan ZM, Qian YK, Zhou W. Malicious code classification method based on deep forest. Ruan Jian Xue Bao/Journal of Software, 2020, 31(5): 1454–1464 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5660.htm
    [28] Zhou TC, Sun XB, Xia X, Li B, Chen X. Improving defect prediction with deep forest. Information and Software Technology, 2019, 114: 204–216. [doi: 10.1016/j.infsof.2019.07.003]
    [29] Xie RL, Cui ZQ, Jia MH, Wen Y, Hao BS. Testing coverage criteria for deep forests. In: Proc. of the 6th Int’l Conf. on Dependable Systems and Their Applications. Harbin: IEEE, 2020. 513–514.
    Related
    Cited by
    Comments
    Comments
    分享到微博
    Submit
Get Citation

崔展齐,谢瑞麟,陈翔,刘秀磊,郑丽伟. DeepRanger:覆盖制导的深度森林测试方法.软件学报,2023,34(5):2251-2267

Copy
Share
Article Metrics
  • Abstract:752
  • PDF: 2329
  • HTML: 1649
  • Cited by: 0
History
  • Received:September 16,2020
  • Revised:January 15,2021
  • Online: September 16,2022
  • Published: May 06,2023
You are the first2044916Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address:4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code:100190
Phone:010-62562563 Fax:010-62562533 Email:jos@iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063