回归测试中的测试用例优先排序技术述评
作者:
基金项目:

国家自然科学基金(60873027, 61202006); 国家高技术研究发展计划(863)(2006AA01Z177); 国家重点基础研究发展计划(973)(2009CB320705); 江苏省高校自然科学研究项目(12KJB520014); 江苏省研究生培养创新工程(CXZZ120935); 南通市应用研究计划(BK2012023); 南京大学计算机软件新技术国家重点实验室开放课题(KFKT2012B29)


Survey of Test Case Prioritization Techniques for Regression Testing
Author:
  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [84]
  • |
  • 相似文献 [20]
  • |
  • 引证文献
  • | |
  • 文章评论
    摘要:

    测试用例优先排序(test case prioritization,简称TCP)问题是回归测试研究中的一个热点.通过设定特定排序准则,对测试用例进行排序以优化其执行次序,旨在最大化排序目标,例如最大化测试用例集的早期缺陷检测速率.TCP 问题尤其适用于因测试预算不足以致不能执行完所有测试用例的测试场景.首先对TCP问题进行描述,并依次从源代码、需求和模型这3 个角度出发对已有的TCP 技术进行分类;然后对一类特殊的TCP 问题(即测试资源感知的TCP 问题)的已有研究成果进行总结;随后依次总结实证研究中常用的评测指标、评测数据集和缺陷类型对实证研究结论的影响;接着依次介绍TCP 技术在一些特定测试领域中的应用,包括组合测试、事件驱动型应用测试、Web 服务测试和缺陷定位等;最后对下一步工作进行展望.

    Abstract:

    Test case prioritization (TCP) issue is a hot research topic in regression testing research. This method tries to optimize the execution schedule based on a specific prioritization criterion. The purpose of the TCP techniques is to maximize a specific prioritization objective, such as the early fault detection rate of the original test suite. This technique is especially applied to some testing scenarios, for example testing resource is limited for executing all the test cases. This paper first describes the issue of TCP and classifies the existing TCP techniques into three categories: source code, requirement, and model. The paper secondly formulates a specific TCP issue (i.e., resource-aware TCP issue) and summarizes its research work. The paper finally summarizes commonly-used evaluation metrics and subjects in experimental studies, and empirical result affection of different fault injection types. The paper fourthly summarizes the application of TCP in some specific testing domains, such as combinatorial testing, event-driven applications testing, fault localization, and Web services testing and discusses some future work of the TCP issue.

    参考文献
    [1] Beizer B. Software Testing Techniques. New York: Van No strand Reinhold, 1990.
    [2] Leung H, White L. Insights into regression testing. In: Proc. of the Int’l Conf. on Software Maintenance. ACM Press, 1989. 60-69.[doi: 10.1109/ICSM.1989.65194]
    [3] Rothermel G, Untch RJ, Chu C. Prioritizing test cases for regression testing. IEEE Trans. on Software Engineering, 2001,27(10):929-948. [doi: 10.1109/32.962562]
    [4] Yoo S, Harman M. Regression testing minimization, selection and prioritization: A survey. Software Testing, Verification &Reliability, 2012,22(2):67-120. [doi: 10.1002/stvr.430]
    [5] Harrold M, Orso A. Retesting software during development and maintenance. In: Proc. of the Frontiers of Software Maintenance.IEEE Press, 2008. 99-108. [doi: 10.1109/FOSM.2008.4659253]
    [6] Wong W, Horgan J, London S, Agrawal H. A study of effective regression testing in practice. In: Proc. of the Int’l Symp. onSoftware Reliability Engineering. IEEE Press, 1997. 264-274. [doi: 10.1109/ISSRE.1997.630875]
    [7] Elbaum S, Malishevsky AG, Rothermel G. Prioritizing test cases for regression testing. In: Proc. of the Int’l Symp. on SoftwareTesting and Analysis. ACM Press, 2000. 102-112. [doi: 10.1145/347324.348910]
    [8] Rothermel G, Untch RH, Chu C, Harrold MJ. Test case prioritization: An empirical study. In: Proc. of the Int’l Conf. on SoftwareMaintenance. IEEE Press, 1999. 179-188. [doi: 10.1109/ICSM.1999.792604]
    [9] Elbaum S, Malishevsky AG, Rothermel G. Test case prioritization: A family of empirical studies. IEEE Trans. on SoftwareEngineering, 2002,28(2):159-182. [doi: 10.1109/32.988497]
    [10] Jones JA, Harrold MJ. Test-Suite reduction and prioritization for modified condition/decision coverage. IEEE Trans. on SoftwareEngineering, 2003,29(3):195-209. [doi: 10.1109/TSE.2003.1183927]
    [11] Do H, Rothermel G, Kinneer A. Empirical studies of test case prioritization in a JUnit testing environment. In: Proc. of the Int’lSymp. on Software Reliability Engineering. IEEE Press, 2004. 113-124. [doi: 10.1109/ISSRE.2004.18]
    [12] Do H, Rothermel G, Kinneer A. Prioritizing JUnit test cases: An empirical assessment and cost-benefits analysis. EmpiricalSoftware Engineering, 2006,11(1):33-70. [doi: 10.1007/s10664-006-5965-8]
    [13] Zhang L, Zhou J, Hao D, Zhang L, Mei H. Prioritizing JUnit test cases in absence of coverage information. In: Proc. of the Int’lConf. on Software Maintenance. IEEE Press, 2009. 19-28. [doi: 10.1109/ICSM.2009.5306350]
    [14] Mei H, Hao D, Zhang LM, Zhang L, Zhou J, Rothermel G. A static approach to prioritizing JUnit test cases. IEEE Trans. onSoftware Engineering, 2012,38(6):1258-1275. [doi: 10.1109/TSE.2011.106]
    [15] Li Z, Harman M, Hierons RM. Search algorithms for regression test case prioritization. IEEE Trans. on Software Engineering,2007,33(4):225-237. [doi: 10.1109/TSE.2007.38]
    [16] Jiang B, Zhang ZY, Chan WK, Tse TH. Adaptive random test case prioritization. In: Proc. of the Int’l Conf. on AutomatedSoftware Engineering. IEEE Press, 2009. 233-244. [doi: 10.1109/ASE.2009.77]
    [17] Voas J. PIE: A dynamic failure-based technique. IEEE Trans. on Software Engineering, 1992,18(8):717-727. [doi: 10.1109/32.153381]
    [18] Jeffrey D, Gupta N. Test case prioritization using relevant slices. In: Proc. of the Annual Int’l Computer Software and ApplicationsConf. IEEE Press, 2006. 411-420. [doi: 10.1109/COMPSAC.2006.80]
    [19] Jeffrey D, Gupta N. Experiments with test case prioritization using relevant slices. Journal of System and Software, 2008,81(2):196-221. [doi: 10.1016/j.jss.2007.05.006]
    [20] Elbaum S, Malishevsky A, Rothermel G. Incorporating varying test costs and fault severities into test case prioritization. In: Proc.of the Int’l Conf. on Software Engineering. IEEE Press, 2001. 329-338. [doi: 10.1109/ICSE.2001.919106]
    [21] Ma ZK, Zhao JJ. Test case prioritization based on analysis of program structure. In: Proc. of the Asia-Pacific Software EngineeringConf. IEEE Press, 2008. 471-478. [doi: 10.1109/APSEC.2008.63]
    [22] Elbaum S, Rothermel G, Kanduri S, Malishevsky AG. Selecting a cost-effective test case prioritization technique. Software QualityControl, 2004,12(3):185-210. [doi: 10.1023/B:SQJO.0000034708.84524.22]
    [23] Arafeen M, Do H. Adaptive regression testing strategy: An empirical study. In: Proc. of the Int’l Symp. on Software ReliabilityEngineering. IEEE Press, 2011. 130-139. [doi: 10.1109/ISSRE.2011.29]
    [24] Li S, Bian N, Chen Z, You D, He Y. A simulation study on some search algorithms for regression test case prioritization. In: Proc.of the Int’l Conf. on Quality Software. 2010. 72-81. [doi: 10.1109/QSIC.2010.15]
    [25] Mirarab S, Tahvildari L. A prioritization approach for software test cases based on bayesian networks. In: Proc. of the Int’l Conf.on Fundamental Approaches to Software Engineering. Springer-Verlag, 2007. 276-290. [doi: 10.1007/978-3-540-71289-3_22]
    [26] Mirarab S, Tahvildari L. An empirical study on Bayesian network-based approach for test case prioritization. In: Proc. of the Int’lConf. on Software Testing, Verification, and Validation. IEEE Press, 2008. 278-287. [doi: 10.1109/ICST.2008.57]
    [27] Leon D, Podgurski A. A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases. In:Proc. of the Int’l Symp. on Software Reliability Engineering. IEEE Press, 2003. 442-453.
    [28] Carlson R, Hyunsook D, Denton A. A clustering approach to improving test case prioritization: An industrial case study. In: Proc.of the Int’l Conf. on Software Maintenance. IEEE Press, 2011. 382-391. [doi: 10.1109/ICSM.2011.6080805]
    [29] Huang YC, Peng KL, Huang CY. A history-based cost-cognizant test case prioritization technique in regression testing. Journal ofSystem and Software, 2012,85(3):626-637. [doi: 10.1016/j.jss.2011.09.063]
    [30] Tonella P, Avesani P, Susi A. Using the case-based ranking methodology for test case prioritization. In: Proc. of the Int’l Conf. onSoftware Maintenance. IEEE Press, 2006. 123-133. [doi: 10.1109/ICSM.2006.74]
    [31] Yoo S, Harman M, Tonella P, Susi A. Clustering test cases to achieve effective and scalable prioritisation incorporating expertknowledge. In: Proc. of the Int’l Symp. on Software Testing and Analysis. ACM Press, 2009. 201-212. [doi: 10.1145/1572272.1572296]
    [32] Srivastava A, Thiagarajan J. Effectively prioritizing tests in development environment. In: Proc. of the Int’l Symp. on SoftwareTesting and Analysis. ACM Press, 2002. 97-106. [doi: 10.1145/566172.566187]
    [33] Sherriff M, Lake M, Williams L. Prioritization of regression tests using singular value decomposition with empirical changerecords. In: Proc. of the Int’l Symp. on Software Reliability. IEEE Press, 2007. 81-90. [doi: 10.1109/ISSRE.2007.20]
    [34] Ramanathan MK, Koyuturk, M, Grama A, Jagannathan S. PHALANX: A graph-theoretic framework for test case prioritization. In:Proc. of the Symp. on Applied Computing. ACM Press, 2008. 667-673. [doi: 10.1145/1363686.1363848]
    [35] Korel B, Tahat L, Harman M. Test prioritization using system models. In: Proc. of the Int’l Conf. on Software Maintenance. IEEEPress, 2005. 559-568. [doi: 10.1109/ICSM.2005.87]
    [36] Korel B, Koutsogiannakis G, Tahat L. Model-Based test prioritization heuristic methods and their evaluation. In: Proc. of the Int’lWorkshop on Advances in Model-Based Testing. ACM Press, 2007. 34-43. [doi: 10.1145/1291535.1291539]
    [37] Korel B, Koutsogiannakis G, Tahat L. Application of system models in regression test suite prioritization. In: Proc. of the Int’lConf. on Software Maintenance. IEEE Press, 2008. 247-256. [doi: 10.1109/ICSM.2008.4658073]
    [38] Kundu D, Sarma M, Samanta D, Mall R. System testing for object-oriented systems with test case prioritization. Software Testing,Verification and Reliability, 2009,19(4):297-333. [doi: 10.1002/stvr.407]
    [39] Srikanth H, Williams L, Osborne J. System test case prioritization of new and regression test cases. In: Proc. of the Int’l Symp. onEmpirical Software Engineering. IEEE Press, 2005. 64-73. [doi: 10.1109/ISESE.2005.1541815]
    [40] Qu B, Nie CH, Xu BW, Zhang XF. Test case prioritization for black box testing. In: Proc. of the Annual Int’l Computer Softwareand Applications Conf. IEEE Press, 2007. 465-474. [doi: 10.1109/COMPSAC.2007.209]
    [41] Qu B, Nie CH, Xu BW. Test case prioritization based on test suite design information. Chinese Journal of Computers, 2008,31(3):431-439 (in Chinese with English abstract).
    [42] Zhang XF, Nie CH, Xu BW, Qu B. Test case prioritization based on varying testing requirement priorities and test case costs. In:Proc. of the Int’l Conf. on Quality Software. IEEE Press, 2007. 15-24. [doi: 10.1109/QSIC.2007.4385476]
    [43] Krishnamoorthi R, Sahaaya Arul Mary SA. Factor oriented requirement coverage based system test case prioritization of new andregression test cases. Information and Software Technology, 2009,51(4):799-808. [doi: 10.1016/j.infsof.2008.08.007]
    [44] Yu YT, Lau MF. Fault-Based test suite prioritization for specification-based testing. Information and Software Technology, 2012,54(2):179-202. [doi: 10.1016/j.infsof.2011.09.005]
    [45] Kim JM, Porter A. A history-based test prioritization technique for regression testing in resource constrained environments. In:Proc. of the Int’l Conf. on Software Engineering. ACM Press, 2002. 119-129. [doi: 10.1145/581339.581357]
    [46] Walcott KR, Soffa ML, Kapfhammer GM, Roos RS. Time-Aware test suite prioritization. In: Proc. of the Int’l Symp. on SoftwareTesting and Analysis. ACM Press, 2006. 1-12. [doi: 10.1145/1146238.1146240]
    [47] Alspaugh S, Walcott KR, Belanich M, Kapfhammer GM, Soffa ML. Efficient time-aware prioritization with knapsack solvers. In:Proc. of the Int’l Workshop on Empirical Assessment of Software Engineering Languages and Technologies. ACM Press, 2007.13-18. [doi: 10.1145/1353673.1353676]
    [48] Zhang L, Hou SS, Guo C, Xie T, Mei H. Time-Aware test-case prioritization using integer linear programming. In: Proc. of theInt’l Symp. on Software Testing and Analysis. ACM Press, 2009. 213-224. [doi: 10.1145/1572272.1572297]
    [49] You DJ, Chen ZY, Xu BW, Luo B, Zhang C. An empirical study on the effectiveness of time-aware test case prioritizationtechniques. In: Proc. of the Symp. on Applied Computing. ACM Press, 2011. 1451-1456. [doi: 10.1145/1982185.1982497]
    [50] Do H, Mirarab S, Tahildari L, Rothermel G. An empirical study of the effect of time constraints on the cost-benefits of regressiontesting. In: Proc. of the Int’l Symp. on Foundations of Software Engineering. ACM Press, 2008. 71-82. [doi: 10.1145/1453101.1453113]
    [51] Do H, Mirarab S, Tahildari L, Rothermel G. The effects of time constraints on test case prioritization: A series of controlledexperiments. IEEE Trans. on Software Engineering, 2010,36(5):593-617. [doi: 10.1109/TSE.2010.58]
    [52] Do H, Elbaum S, Rothermel G. Supporting controlled experimentation with testing techniques: an infrastructure and its potentialimpact. Empirical Software Engineering, 2005,10(4):405-435. [doi: 10.1007/s10664-005-3861-2]
    [53] Hutchins M, Foster H, Goradia T, Ostrand T. Experiments on the effectiveness of dataflow- and control- flow-based test adequacycriteria. In: Proc. of the Int’l Conf. on Software Engineering. ACM Press, 1994. 191-200.
    [54] Andrews JH, Briand LC, Labiche Y. Is mutation an appropriate tool for testing experiments? In: Proc. of the Int’l Conf. onSoftware Engineering. ACM Press, 2005. 402-411. [doi: 10.1145/1062455.1062530]
    [55] Do H, Rothermel G. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans. onSoftware Engineering, 2006,32(9):733-752. [doi: 10.1109/TSE.2006.92]
    [56] Bryce RC, Colbourn CJ. Test prioritization for pairwise interaction coverage. In: Proc. of the Int’l Workshop on Advances inModel-based Testing. ACM Press, 2005. 1-7. [doi: 10.1145/1083274.1083275]
    [57] Bryce RC, Colbourn CJ. Prioritized interaction testing for pair-wise coverage with seeding and constraints. Information andSoftware Technology, 2006,48(10):960-970. [doi: 10.1016/j.infsof.2006.03.004]
    [58] Wang ZY, Xu BW, Chen L, Chen ZY. Cost-Effective combinatorial test case prioritization for varying combination weights. In:Proc. of the Int’l Conf. on Software & Knowledge Engineering. Redwood: Knowledge Systems Institute Graduate School, 2010.273-278.
    [59] Wang ZY, Chen L, Xu BW, Huang Y. Cost-Cognizant combinatorial test case prioritization. Int’l Journal of Software Engineeringand Knowledge Engineering, 2011,21(6):829-854. [doi: 10.1142/S0218194011005499]
    [60] Qu X, Cohen MB, Woolf K. Combinatorial interaction regression testing: A study of test case generation and prioritization. In:Proc. of the Int’l Conf. on Software Maintenance. IEEE Press, 2007. 255-264. [doi: 10.1109/ICSM.2007.4362638]
    [61] Qu X, Cohen MB, Rothermel G. Configuration-Aware regression testing: An empirical study of sampling and prioritization. In:Proc. of the Int’l Symp. on Software Testing and Analysis. ACM Press, 2008. 75-86. [doi: 10.1145/1390630.1390641]
    [62] Srikanth H, Cohen MB, Qu X. Reducing field failures in system configurable software: Cost-based prioritization. In: Proc. of theInt’l Conf. on Software Reliability Engineering. IEEE Press, 2009. 61-70. [doi: 10.1109/ISSRE.2009.26]
    [63] Bryce RC, Memon AM. Test suite prioritization by interaction coverage. In: Proc. of the Workshop on Domain SpecificApproaches to Software Test Automation. ACM Press, 2007. 1-7. [doi: 10.1145/1294921.1294922]
    [64] Huang CY, Chang JR, Chang YH. Design and analysis of GUI test-case prioritization using weighted-based methods. Journal ofSystem and Software, 2010,83(4):646-659. [doi: 10.1016/j.jss.2009.11.703]
    [65] Sampath S, Bryce RC, Viwanath G, Kandimalla V, Koru AG. Prioritizing user-session-based test cases for Web applicationstesting. In: Proc. of the Int’l Conf. on Software Testing, Verification, and Validation. IEEE Press, 2008. 141-150. [doi: 10.1109/ICST.2008.42]
    [66] Garg D, Datta A. Test case prioritization due to database changes in Web applications. In: Proc. of the Int’l Conf. on SoftwareTesting, Verification and Validation. IEEE Press, 2012. 726-730. [doi: 10.1109/ICST.2012.163]
    [67] Bryce RC, Sampath S, Memon AM. Developing a single model and test prioritization strategies for event-driven software. IEEETrans. on Software Engineering, 2011,37(1):48-64. [doi: 10.1109/TSE.2010.12]
    [68] Hou SS, Zhang L, Xie T, Mei H, Sun JS. Applying interface-contract mutation in regression testing of component-based software.In: Proc. of the Int’l Conf. on Software Maintenance. IEEE Press, 2007. 174-183. [doi: 10.1109/ICSM.2007.4362630]
    [69] Hou SS, Zhang L, Xie T, Sun JS. Quota-Constrained test-case prioritization for regression testing of service-centric systems. In:Proc. of the Int’l Conf. on Software Maintenance. IEEE Press, 2008. 257-266. [doi: 10.1109/ICSM.2008.4658074]
    [70] Mei LJ, Chan WK, Tse TH, Merkel RG. Tag-Based techniques for black-box test case prioritization for service testing. In: Proc. ofthe Int’l Conf. on Quality Software. IEEE Press, 2009. 21-30. [doi: 10.1109/QSIC.2009.12]
    [71] Mei LJ, Chan WK, Tse TH, Merkel RG. XML-Manipulating test case prioritization for XML-manipulating services. Journal ofSystem and Software, 2011,84(4):603-619. [doi: 10.1016/j.jss.2010.11.905]
    [72] Mei LJ, Zhang ZY, Chan WK, Tse TH. Test case prioritization for regression testing of service-oriented business applications. In:Proc. of the Int’l Conf. on World Wide Web. ACM Press, 2009. 901-910. [doi: 10.1145/1526709.1526830]
    [73] Chen L, Wang ZY, Xu L, Lu HM, Xu BW. Test case prioritization for Web service regression testing. In: Proc. of the Int’l Symp.on Service Oriented System Engineering. IEEE Press, 2010. 173-178. [doi: 10.1109/SOSE.2010.27]
    [74] Nguyen CD, Marchetto A, Tonella P. Test case prioritization for audit testing of evolving Web services using information retrievaltechniques. In: Proc. of the Int’l Conf. on Web Services. IEEE Press, 2011. 636-643. [doi: 10.1109/ICWS.2011.75]
    [75] Zhai K, Jiang B, Chan W, Tse T. Taking advantage of service selection: A study on the testing of location-based Web servicesthrough test case prioritization. In: Proc. of the Int’l Conf. on Web Services. IEEE Press, 2010. 211-218. [doi: 10.1109/ICWS.2010.98]
    [76] Zhai K, Chan WK. Point-of-Interest aware test case prioritization: Methods and experiments. In: Proc. of the Int’l Conf. on QualitySoftware. IEEE Press, 2010. 449-456. [doi: 10.1109/QSIC.2010.63]
    [77] Jiang B, Chan WK. On the integration of test adequacy, test case prioritization, and statistical fault localization. In: Proc. of theInt’l Conf. on Quality Software. IEEE Press, 2010. 377-384. [doi: 10.1109/QSIC.2010.64]
    [78] Jiang B, Chan WK, Tse TH. On practical adequate test suites for integrated test case prioritization and fault localization. In: Proc.of the Int’l Conf. on Quality Software. IEEE Press, 2011. 21-30. [doi: 10.1109/QSIC.2011.37]
    [79] Jiang B, Zhang ZY, Chan WK, Tse TH, Chen TY. How well does test case prioritization integrate with statistical fault localization?Information Software Technology, 2012,54(7):739-758. [doi: 10.1016/j.infsof.2012.01.006]
    [80] Gonzalez-Sanchez A, Piel E, Gross H, van Gemund A. Prioritizing tests for software fault localization. In: Proc. of the Int’l Conf.on Quality Software. IEEE Press, 2010. 42-51. [doi: 10.1109/QSIC.2010.28]
    [81] Gonzalez-Sanchez A, Abreu R, Gross H, Gemund A. Prioritizing tests for fault localization through ambiguity group reduction. In:Proc. of the Int’l Conf. on Automated Software Engineering. IEEE Press, 2011. 83-92. [doi: 10.1109/ASE.2011.6100153]
    [82] Kim S, Baik J. An effective fault aware test case prioritization by incorporating a fault localization technique. In: Proc. of the Int’lSymp. on Empirical Software Engineering and Measurement. ACM Press, 2010. 1-10. [doi: 10.1145/1852786.1852793]
    [83] Just R, Kapfhammer GM, Schweiggert F. Using non-redundant mutation operators and test suite prioritization to achieve efficientand scalable mutation analysis. In: Proc. of the Int’l Symp. on Software Reliability Engineering. IEEE Press, 2012. 11-20. [doi: 10.1109/ISSRE.2012.31]
    [84] Staats M, Loyola P, Rothermel G. Oracle-Centric test case prioritization. In: Proc. of the Int’l Symp. on Software ReliabilityEngineering. IEEE Press, 2012. 311-320. [doi: 10.1109/ISSRE.2012.13]
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

陈翔,陈继红,鞠小林,顾庆.回归测试中的测试用例优先排序技术述评.软件学报,2013,24(8):1695-1712

复制
分享
文章指标
  • 点击次数:9486
  • 下载次数: 13816
  • HTML阅读次数: 0
  • 引用次数: 0
历史
  • 收稿日期:2013-01-21
  • 最后修改日期:2013-03-29
  • 在线发布日期: 2013-05-23
文章二维码
您是第20233571位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号