基于规则与学习的变异技术对比研究
作者:
作者简介:

贡志豪(2000—),男,博士生,CCF学生会员,主要研究领域为软件工程;陈逸洲(1996—),男,博士生,CCF学生会员,主要研究领域为软件测试,区块链安全和深度学习;陈俊洁(1992—),男,博士,副教授,CCF高级会员,主要研究领域为软件分析与测试;郝丹(1979—),女,博士,教授,博士生导师,CCF杰出会员,主要研究领域为软件测试.

通讯作者:

郝丹,E-mail:haodan@pku.edu.cn

基金项目:

国家重点研发计划(2022YFB4501902);国家自然科学基金(62232001)


Comparison Research on Rule-based and Learning-based Mutation Techniques
Author:
  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [31]
  • |
  • 相似文献 [20]
  • | | |
  • 文章评论
    摘要:

    变异测试是一种有效的软件测试技术, 通过生成变异体来模拟软件缺陷, 帮助提升现有测试套件的缺陷检测能力. 变异体的质量对于变异测试的有效性具有显著影响. 传统的变异测试方法通常采用人工设计的基于语法规则的变异算子生成变异体, 并已取得一定的研究成果. 近年来, 许多研究开始结合深度学习技术, 通过学习开源项目历史代码生成变异体. 目前, 该新方法在变异体生成方面取得了初步的成果. 基于语法规则和基于学习的两种变异技术, 其机理不同, 但其目标均是通过生成变异体来提高测试套件的缺陷检测能力, 因此, 全面比较这两种变异技术对于变异测试及其下游任务至关重要. 针对这一问题, 设计实现了一项针对基于语法规则和基于学习的变异技术的实证研究, 旨在了解不同机理的变异技术在变异测试任务上的性能以及生成的变异体在程序语义上的差异性. 具体地, 以Defect4J v1.2.0数据集为实验对象, 比较以MAJOR和PIT为代表的基于语法规则的变异技术和以DeepMutation、μBERT和LEAM为代表的基于深度学习的变异技术. 实验结果表明: 基于规则与学习的变异技术均可有效支持变异测试实践, 但MAJOR的测试效果最优, 能够检测85.4%的真实缺陷. 在语义表示上, MAJOR具有最强的语义代表能力, 基于其构造的测试套件能够杀死其余变异技术生成的超过95%占比的变异体. 在缺陷表征上, 两类技术均具有独特性.

    Abstract:

    Mutation testing is an effective software testing technique. It helps improve the defect detection capability of an existing test suite by generating mutants that simulate software defects. The quality of mutants has a significant impact on the effectiveness of mutation testing. The traditional mutation testing approach typically employs manually designed syntactic rule-based mutation operators to generate mutants, and has achieved some academic success. In recent years, many studies have started to incorporate deep learning techniques to generate mutants by learning historical code from open source projects. This new approach has achieved preliminary progress in mutant generation. A comprehensive comparison of the two mutation techniques, i.e. rule-based and learning-based, which have different mechanisms but both aim to improve the defect detection capability of the test suite by mutation, is crucial for mutation testing and its downstream tasks. To handle the problem, this study designs and implements an empirical study of rule-based and learning-based mutation techniques, aiming to understand the performance of mutation techniques with different mechanisms on the task of mutation testing, as well as the variability of the generated mutants in terms of program semantics. Specifically, this study uses the Defect4J v1.2.0 dataset to compare the syntactic rule-based mutation techniques represented by MAJOR and PIT with the deep learning-based mutation techniques represented by DeepMutation, μBERT, and LEAM. The experimental results show that both rule-based and learning-based mutation techniques can effectively support mutation testing practices, but MAJOR has the best testing performance and is able to detect 85.4% of real defects. In terms of semantic representation, MAJOR has the strongest semantic representation capability, and its constructed test suite is able to kill more than 95% of the mutants generated by other mutation techniques. In terms of defect representation, both types of techniques are unique.

    参考文献
    [1] Ammann P, Offutt J. Introduction to Software Testing. Cambridge University Press, 2016.
    [2] Moore I. Jester-a JUnit test tester. In: Proc. of the 2nd XP. 2001. 84-87.
    [3] Ma YS, Offutt J, Kwon YR. MuJava: An automated class mutation system. Software Testing, Verification and Reliability, 2005, 15(2): 97-133.
    [4] Irvine SA, Pavlinic T, Trigg L, et al. Jumble Java byte code to measure the effectiveness of unit tests. In: Proc. of the Testing: Academic and Industrial Conf. on Practice and Research Techniques-MUTATION (TAICPART-MUTATION 2007). IEEE, 2007. 169-175.
    [5] Schuler D, Zeller A. Javalanche: Efficient mutation testing for Java. In: Proc. of the 7th Joint Meeting of the European Software Engineering Conf. and the ACM SIGSOFT Symp. on The Foundations of Software Engineering. 2009. 297-298.
    [6] Just R. The major mutation framework: Efficient and scalable mutation analysis for Java. In: Proc. of the 2014 Int'l Symp. on Software Testing and Analysis. 2014. 433-436.
    [7] Coles H, Laurent T, Henard C, et al. PIT: A practical mutation testing tool for Java. In: Proc. of the 25th Int'l Symp. on Software Testing and Analysis. 2016. 449-452.
    [8] Papadakis M, Kintis M, Zhang J, et al. Mutation testing advances: An analysis and survey. Advances in Computers, 2019, 112: 275-378.
    [9] Papadakis M, Le Traon Y. Metallaxis-FL: Mutation-based fault localization. Software Testing, Verification and Reliability, 2015, 25(5-7): 605-628.
    [10] Zhang J, Wang Z, Zhang L, et al. Predictive mutation testing. In: Proc. of the 25th Int'l Symp. on Software Testing and Analysis. 2016. 342-353.
    [11] Ghanbari A, Benton S, Zhang L. Practical program repair via bytecode mutation. In: Proc. of the 28th ACM SIGSOFT Int'l Symp. on Software Testing and Analysis. 2019. 19-30.
    [12] Tufano M, Watson C, Bavota G, et al. Learning how to mutate source code from bug-fixes. In: Proc. of the 2019 IEEE Int'l Conf. on Software Maintenance and Evolution (ICSME). IEEE, 2019. 301-312.
    [13] Degiovanni R, Papadakis M. μBERT: Mutation testing using pre-trained language models. In: Proc. of the 2022 IEEE Int'l Conf. on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2022. 160-169.
    [14] Tian Z, Chen J, Zhu Q, et al. Learning to construct better mutation faults. In: Proc. of the 37th IEEE/ACM Int'l Conf. on Automated Software Engineering. 2022. 1-13.
    [15] Amalfitano D, Paiva ACR, Inquel A, et al. How do Java mutation tools differ? Communications of the ACM, 2022, 65(12): 74-89.
    [16] 王赞, 闫明, 刘爽, 陈俊洁, 张栋迪, 吴卓, 陈翔. 深度神经网络测试研究综述. 软件学报, 2020, 31(5): 1255-1275. http://www.jos.org.cn/1000-9825/5951.htm [doi: 10.13328/j.cnki.jos.005951]
    Wang Z, Yan M, Liu S, Chen JJ, Zhang DD, Wu Z, Chen X. Survey on testing of deep neural networks. Ruan Jian Xue Bao/Journal of Software, 2020, 31(5): 1255-1275(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5951.htm [doi: 10. 13328/j.cnki.jos.005951]
    [17] Jia Y, Harman M. Higher order mutation testing. Information and Software Technology, 2009, 51(10): 1379-1393.
    [18] Gopinath R, Jensen C, Groce A. The theory of composite faults. In: Proc. of the 2017 IEEE Int'l Conf. on Software Testing, Verification and Validation (ICST). IEEE, 2017. 47-57.
    [19] Offutt AJ, Untch RH. Mutation 2000: Uniting the orthogonal. In: Mutation Testing for the New Century. 2001. 34-44.
    [20] Ammann P, Delamaro ME, Offutt J. Establishing theoretical minimal sets of mutants. In: Proc. of the 7th IEEE Int'l Conf. on Software Testing, Verification and Validation. IEEE, 2014. 21-30.
    [21] Zhang L, Marinov D, Zhang L, et al. Regression mutation testing. In: Proc. of the 2012 Int'l Symp. on Software Testing and Analysis. 2012. 331-341.
    [22] Wang B, Xiong Y, Shi Y, et al. Faster mutation analysis via equivalence modulo states. In: Proc. of the 26th ACM SIGSOFT Int'l Symp. on Software Testing and Analysis. 2017. 295-306.
    [23] 姚毅文, 姜淑娟, 薄莉莉. 基于变异测试的错误定位研究进展. 计算机工程与应用, 2019, 55(20): 1-12.
    Yao YW, Jiang SJ, Bo LL. Survey of mutation-based fault location. Computer Engineering and Applications, 2019, 55(20): 1-12(in Chinese with English abstract).
    [24] Zhang J, Zhu M, Hao D, et al. An empirical study on the scalability of selective mutation testing. In: Proc. of the 25th IEEE Int'l Symp. on Software Reliability Engineering. IEEE, 2014. 277-287.
    [25] Beller M, Wong CP, Bader J, et al. What it would take to use mutation testing in industry—A study at Facebook. In: Proc. of the 43rd IEEE/ACM Int'l Conf. on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2021. 268-277.
    [26] DeMillo RA, Lipton RJ, Sayward FG. Hints on test data selection: Help for the practicing programmer. Computer, 1978, 11(4): 34-41.
    [27] Jia Y, Harman M. An analysis and survey of the development of mutation testing. IEEE Trans. on Software Engineering, 2010, 37(5): 649-678.
    [28] Offutt AJ. Investigations of the software testing coupling effect. ACM Trans. on Software Engineering and Methodology (TOSEM), 1992, 1(1): 5-20.
    [29] Offutt A. The coupling effect: Fact or fiction. ACM SIGSOFT Software Engineering Notes, 1989, 14(8): 131-1愴渰搮?灢牲椾浛攳‰灝愠瑊桵?捴漠癒攬爠慊条敬???渠??倠牉潮捯??潭晴?瑥桶敡???????湡瑬?氠??潥渠晭??潡湮?即漠晡琠睶慡牬敩?吠敳獵瑢楳湴杩??噴敥爠楦景楲挠慲瑥楡潬渠??慵湬摴?嘠慩汮椠摳慯瑦楴潷湡?坥漠牴步獳桴潩灮獧?????????っ????㈠?と?㈠有???戠牁?孍?こ嵉??潏灆楔渠慉瑮桴?剬???敭湰献攠湯?????牮潤捡整?????潯摦攠?捯潦癴敷牡慲来攠?普潧物?獥略楲瑩敮?攮瘠愲氰由愴琮椠漶渵?戭礶?搵攮瘼敢汲漾灛攳爱獝???湰??偡牴潨挠??漠晊?瑮桳敥???琬栠??湯瑣?氠??漠湍晵??潴湩?卮潳昺琠睈慯牷攠??湯杳楥渠敡敲牥椠湴杨??㈠ぴ???????????扴牳?嬠??崺??湲潯穣攮洠瑯獦攠癴慨???‵?潨氠浉故獅?删???漧癬攠牓慹杭数?椠獯?渠潓瑯?獴瑷牡潲湥朠汒祥?捩潡牢物敬汩慴瑹攠摅?睧楩瑮桥?瑲敩獮瑧?猠畉楅瑅故?攠昲昰攱挴琮椠瘱攸渹攭猲猰???湢? ̄偛爳漲捝??潲景?瑮栠敄???瑖桡??湨瑮?汍??潌湩晢??潴渠?匬漠晥瑴眠慡牬攮??湨来椠湣敡敲牥椠湡杮???づ?????????????戭牣?孵??嵴??極湴瑡楮獴????偮愺瀠慐摲慯正椮猠????偨慥瀠愱搱潴灨漠畊汯潩獮????敥瑴?慮汧???漠睆?敵普晤敡捴瑩楯癮敳?慯牦攠?浯畦瑴慷瑡楲潥渠?瑮敧獩瑮楥湥杲?瑮潧漮氠猲???渮?攵洱瀱椭爵椲挲愮氼?慲渾慛氳礳獝椠獋?潡普??慲瘠慁?洠畋瑯慹瑵楮潣湵?瑁攬猠瑐楡湰条?瑡潫潩汳猠?眬椠瑥桴?浡慬渮甠慉求?慒渺愠求祵獧椭獲?慰湯摲?爭敤慲汩?晥慮甠汦瑡獵???浩灮楪牥楣捴慩汯?匮漠晁瑃睍愠牔敲??湳朮椠湯敮攠牓楯湦杴???づㄠ??????????????????扴牨?孤??嵯?佹樬搠愲渰椲挳???′?愲爩机???″?栮愼湢晲椾牛″???敌瑩?愠汋??卋祯湹瑵慮捣瑵椠捁?瘠敋物獭甠獄?猠敥浴愠湡瑬椮挠?獂楡浲椺氠慒牥楶瑩祳?潴晩?慧爠瑴楥晭楰捬楡慴汥?慢湡摳?牤攠慡汵?景慭畡汴瑥獤?楰湲?浧畲瑡慭琠楲潥湰?瑩敲献琠楉湮机?獐瑲畯摣椮攠獯??????′吸牴慨渠獁??漠湓?升潓晏瑆睔愠牉敮??湬朠楓湹敭数爮椠湯杮???て?????????????㈠????????批牳?孳??崲‰伱樹搮愠渳椱挭?????桲愾湛昳椵牝?????愠牚本????攠瑄?愠汔??佧渠?挬漠浥灴愠牡楬渮朠?浯畤瑥慂瑅楒潔渺?瑁攠獰瑲楥渭杴?瑡潩潮汥獤?瑭桯牤潥畬朠桦?汲攠慰牲湯楧湲条?扭慩獮敧搠?浮畤琠慮湡瑴?獲敡汬攠捬瑡楮潧湵???湳??偉牮漺挠??潯晣?琠桯敦??瑨桥???????????湦琠?汨??潁湳晳??潩湡??畯瑮漠浦慯瑲椠潃湯?潰晵?卡潴晩瑯睮慡牬攠?呩敮獧瑵???卩呣?㈠〨?????㈠?日???戮爠?嬰??崮?吱电昳愶渭漱‵????楢浲款潛″???坃慨湥杫?卭??敔琬?慐污???敡敫灩浳甠瑍愬琠楌潥渠????湮攠留爬愠汥?洠畡瑬愮琠楁潮渠?瑭潰潩汲???湬??側牵潤捹??潮映?瑵桴敡???湮搬???????????湡瑮?氠??潡湮晣??潣湯?卥潲晡瑧睥愠牦敡??湴朠楲湥敶敥牬楡湴杩???潴浨灡慴渠楡潶湯?偤牳漠捴敨敥搠極湮杲獥???ぢ?づ??????㈠??扯牧?孡??嵡??畵獭瑰?剩???愠汉慮氺椠?????爠湯獦琠????″?整晨攠捉瑅獅??????摉慮瑴愧扬愠獃敯?潦昮?敯确椠獓瑯楦湴杷?晲慥甠汅瑮獧?瑮潥?敲湩慮执氠攨?捃潓湅琩爮漠汉汅故摅?琠攲猰琱椷渮朠‵猹琷甭搶椰攸献?晢潲爾??愷癝愠?灡牰潡杤牡慫浩獳???渠??偩牮漠捄??潙景?琠桓攬??ぴㄠ???渠瑁?汥?卭祵浴灡??潯湮?即潣景瑲睥慳爠散?呲敲獥瑬楡湴来?愠湷摩??渠慲汥祡獬椠獦???ぴㄠ????????????戠牬?孲??崠?偣慡灬慥搠慥歭楰獩?????攠湳慴牵摤?????慴牨浥愠湲????敩瑯?慳汨??吠桢牥整慷瑥獥?琠潭?瑴桡敮?癳愠污楮摤椠瑲祥?潬映?浡畵瑬慴瑳椮漠湉?戺愠獐敲摯?琮攠獯瑦?慴獨獥攠猴猰浴敨渠瑉???湬??偯牮潦挮??潮映?瑯桦整???瑥栠??湧瑩?汥?卲祩浮灧??漲渰?匸漮映琵眳愷爭攵?吸攮猼瑢楲渾杛″愸湝搠??湥慮氠祙獔椬猠???ど?????????????扡牭?孬??崠?圬椠汥捴漠硡潬渮?????湳摩楴癩楮摧甠慴汨?挠潲浥灬慡牴楩獯潮湳獨?潰映?来牴潷略灥敮搠?摡慵瑬慴?扤祥?牥慣湴歩楯湮本?浴敥瑳桴漠摡獤???潡畣特渠慣汲?潴晥??捡漬渠潡浮楤挠??湳瑴漠浳潥汴漠杳祩????????????????????????扨爠?孅??崯?噃慍爠杉桮慴?????敮汦愮渠敯祮???????捴牥楤琠楓煯畦整?慡湲摥?楅浮灧物潮癥敥浲敩湮瑧?漠昲‰琲栰攮????挭漲洴洹漮渼?汲愾湛朳甹慝朠敌?攠晎昬攠捐瑲?獰楨穡敭?獮瑴慲瑩楰獯瑮楧挠獕?漠晏??捵?牴愠睊?愠湁摮?坥潸湰来???潥畮牴湡慬氠?潯晭??摲畩捳慯瑮椠潯湦愠汦?慵湲搠??敩桴愠癴楥潳牴愠汣?卩瑴慥瑲楩獡琺椠捍獵???どはの??????????ひ???????扳牥?嬠? 崀?伀昀昀甀琀琀??????洀甀琀愀琀椀漀渀?挀愀爀漀氀??倀愀猀琀??瀀爀攀猀攀渀琀?愀渀搀?昀甀琀甀爀攀???渀昀漀爀洀愀琀椀漀渀?愀渀搀?匀漀昀琀眀愀爀攀?吀攀挀栀渀漀氀漀最礀??? ???????? ???? ????? ???戀爀?嬀??崀?倀愀瀀愀搀愀欀椀猀?????愀氀攀瘀爀椀猀?一???甀琀漀洀愀琀椀挀?洀甀琀愀琀椀漀渀?琀攀猀琀?挀愀猀攀?最攀渀攀爀愀琀椀漀渀?瘀椀愀?搀礀渀愀洀椀挀?猀礀洀戀漀氀椀挀?攀砀攀挀甀琀椀漀渀???渀??倀爀漀挀??漀昀?琀栀攀???猀琀???????渀琀?氀?匀礀洀瀀??漀渀?匀漀昀琀眀愀爀攀?刀攀氀椀愀戀椀氀椀琀礀??渀最椀渀攀攀爀椀渀最????????? ? ???????? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

贡志豪,陈逸洲,陈俊洁,郝丹.基于规则与学习的变异技术对比研究.软件学报,2024,35(7):3093-3114

复制
分享
文章指标
  • 点击次数:544
  • 下载次数: 2660
  • HTML阅读次数: 892
  • 引用次数: 0
历史
  • 收稿日期:2023-09-11
  • 最后修改日期:2023-10-30
  • 在线发布日期: 2024-01-05
  • 出版日期: 2024-07-06
文章二维码
您是第19728113位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号