Test Set Optimization in Continuous Integration: A Systematic Literature Review
Author:
Affiliation:

Fund Project:

National Natural Science Foundation of China (61432001, 61602450)

  • Article
  • | |
  • Metrics
  • |
  • Reference [57]
  • |
  • Related [20]
  • |
  • Cited by
  • | |
  • Comments
    Abstract:

    Currently, many software applications are rapidly developed and deployed. Rapid iterations are used to implement new requirements while keeping high quality. Continuous integration (CI) is an important activity to make it work. CI tests the code automatically before it will be integrated into the master branch to ensure its quality. The big challenge is how to select an appropriate test case set for continuous integration. Running all test cases will consume a large amount of resource and get feedback in a long cycle, but selecting an improper test case subset will lose some required test, and therefore increase quality risk. The goal of test set optimization is to select an appropriate test case subset which can satisfy the test requirement for each unit or iteration test while reducing the test resource as much as possible. In this paper, a systematic literature review is performed to reveal the current status of CI-test set optimization based on existing literature. More specifically, the related studies published between 2007 and 2017 from four databases of electronic literature are searched to finally select 39 important studies for careful review. Five research questions are presented in order to achieve the above goal, and data from the selected studies are extracted to analyze and answer the questions. The analysis results contribute some useful findings and new challenges for researches and practitioners.

    Reference
    [1] Hilton M, Nelson N, Tunnell T, Marinov D, Dig D. Trade-Offs in continuous integration:Assurance, security, and flexibility. In:Proc. of the Joint Meeting on Foundations of Software Engineering. 2017. 197-207.
    [2] Vasilescu B, Yu Y, Wang H, Devanbu P, Filko V. Quality and productivity outcomes relating to continuous integration in GitHub. In:Proc. of the Joint Meeting on Foundations of Software Engineering. 2015. 805-816.
    [3] Hilton M, Tunnell T, Huang K, Marinov D, Dig D. Usage, costs, and benefits of continuous integration in open-source projects. In:Proc. of the IEEE/ACM Int'l Conf. on Automated Software Engineering. 2016. 426-437.
    [4] Martensson T, Stahl D, Bosch J. Continuous integration impediments in large-scale industry projects. In:Proc. of the IEEE Int'l Conf. on Software Architecture. IEEE, 2017. 169-178.
    [5] Virmani M. Understanding DevOps & bridging the gap from continuous integration to continuous delivery. In:Proc. of the 5th Int'l Conf. on Innovative Computing Technology (INTECH). 2015. 78-82.
    [6] Rathod N, Surve A. Test orchestration a framework for continuous integration and continuous deployment. In:Proc. of the Int'l Conf. on Pervasive Computing. 2015. 1-5.
    [7] Kitchenham BA, Charters S. Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report, EBSE-2007-01, ICSE IEEE Computer Society, 2007. 1051-1052.
    [8] Badampudi D, Wohlin C, Kai P. Experiences from using snowballing and database searches in systematic literature studies. In:Proc. of the Int'l Conf. on Evaluation and Assessment in Software Engineering. 2015. 1-10.
    [9] Kitchenham BA, Dyba T, Jorgensen M. Evidence-Based software engineering. In:Proc. of the Int'l Conf. on Software Engineering. IEEE Computer Society, 2004. 273-281.
    [10] Yoo S, Harman M. Regression testing minimization, selection and prioritization:A survey. Software Testing, Verification and Reliability, 2012,22(2):67-120.
    [11] Kazmi R, Jawawi DNA, Mohamad R, Ghani I. Effective regression test case selection:A systematic literature review. ACM Computing Surveys, 2017,50(2):29.
    [12] Nanda A, Mani S, Sinha S, Harrold MJ, Orso A. Regression testing in the presence of non-code changes. In:Proc. of the 4th IEEE Int'l Conf. on Software Testing, Verification and Validation. 2011. 21-30.
    [13] Qiu D, Li B, Ji S, Leung H. Regression testing of Web service:A systematic mapping study. ACM Computing Surveys, 2015,47(2):21.
    [14] Stratis P, Rajan A. Test case permutation to improve execution time. In:Proc. of the IEEE/ACM Int'l Conf. on Automated Software Engineering. 2016. 45-50.
    [15] Huang CY, Chen CS, Lai CE. Evaluation and analysis of incorporating fuzzy expert system approach into test suite reduction. Information and Software Technology, 2016,79(C):79-105.
    [16] Qu X, Cohen MB, Woolf KM. Combinatorial interaction regression testing:A study of test case generation and prioritization. In:Proc. of the IEEE Int'l Conf. on Software Maintenance. 2007. 255-264.
    [17] Engström E, Runeson P, Skoglund M. A systematic review on regression test selection techniques. Information and Software Technology, 2010,52(1):14-30.
    [18] Catal C, Mishra D. Test case prioritization:A systematic mapping study. Software Quality Journal, 2013,21(3):445-478.
    [19] Soetens QD, Demeyer S, Zaidman A, Pérez J. Change-Based test selection:An empirical evaluation. Empirical Software Engineering, 2016,21(5):1990-2032.
    [20] Blondeau V, Etien A, Anquetil N, Cresson S, Croisy P, Ducasse S. Test case selection in industry:An analysis of issues related to static approaches. Software Quality Journal, 2016, 1-35.
    [21] Spieker H, Gotlieb A, Marijan D, Mossige M. Reinforcement learning for automatic test case prioritization and selection in continuous integration. In:Proc. of the 26th ACM SIGSOFT Int'l Symp. on Software Testing and Analysis. 2017. 12-22.
    [22] Busjaeger B, Xie T. Learning for test prioritization:An industrial case study. In:Proc. of the 24th ACM SIGSOFT Int'l Symp. on Foundations of Software Engineering. 2016. 975-980.
    [23] Elbaum S, Rothermel G, Penix J. Techniques for improving regression testing in continuous integration development environments. In:Proc. of the 22nd ACM SIGSOFT Int'l Symp. on Foundations of Software Engineering. 2014. 235-245.
    [24] Parsai A, Soetens QD, Murgia A, Demeyer S. Considering polymorphism in change-based test suite reduction. In:Proc. of the Int'l Conf. on Agile Software Development. Cham:Springer-Verlag, 2014. 166-181.
    [25] Shi A, Thummalapenta S, Lahiri SK, Bjorner N, Czerwonka J. Optimizing test placement for module-level regression testing. In:Proc. of the IEEE/ACM Int'l Conf. on Software Engineering. IEEE Computer Society, 2017. 689-699.
    [26] Memon A, Gao Z, Nguyen B, Dhanda S, Nickell E, Siemborski R, Micco J. Taming Google-scale continuous testing. In:Proc. of the 39th Int'l Conf. on Software Engineering. 2017. 233-242.
    [27] Marijan D, Liaaen M, Gotlieb A, Sen S, Ieva C. TITAN:Test suite optimization for highly configurable software. In:Proc. of the IEEE Int'l Conf. on Software Testing, Verification and Validation (ICST). 2017. 524-531.
    [28] Ekelund ED, Engstrom E. Efficient regression testing based on test history:An industrial evaluation. In:Proc. of the IEEE Int'l Conf. on Software Maintenance and Evolution. 2015. 449-457.
    [29] Noor TB, Hemmati H. A similarity-based approach for test case prioritization using historical failure data. In:Proc. of the 26th IEEE Int'l Symp. on Software Reliability Engineering (ISSRE). 2015. 58-68.
    [30] Hemmati H, Fang Z, Mantyla MV. Prioritizing manual test cases in traditional and rapid release environments. In:Proc. of the 8th IEEE Int'l Conf. on Software Testing, Verification and Validation (ICST). 2015. 1-10.
    [31] Herzig K, Greiler M, Czerwonka J, Murphy B. The art of testing less without sacrificing quality. In:Proc. of the IEEE/ACM Int'l Conf. on Software Engineering. 2015. 483-493.
    [32] Bach T, Andrzejak A, Pannemans R. Coverage-Based reduction of test execution time:Lessons from a very large industrial project. In:Proc. of the IEEE Int'l Conf. on Software Testing, Verification and Validation Workshops. IEEE, 2017. 3-12.
    [33] Beszédes Á, Gergely T, Schrettner L, Jász J, Langó L, Gyimóthy T. Code coverage-based regression test selection and prioritization in WebKit. In:Proc. of the 28th IEEE Int'l Conf. on Software Maintenance (ICSM). 2012. 46-55.
    [34] Strandberg PE, Sundmark D, Afzal W, Ostrand TJ, Weyuker EJ. Experience report:Automated system level regression test prioritization using multiple factors. In:Proc. of the 27th IEEE Int'l Symp. on Software Reliability Engineering (ISSRE). 2016. 12-23.
    [35] Engström E, Runeson P, Ljung A. Improving regression testing transparency and efficiency with history-based prioritization-an industrial case study. In:Proc. of the 4th IEEE Int'l Conf. on Software Testing, Verification and Validation (ICST). 2011. 367-376.
    [36] Srikanth H, Cashman M, Cohen MB. Test case prioritization of build acceptance tests for an enterprise cloud application:An industrial case study. Journal of Systems and Software, 2016,119:122-135.
    [37] Nguyen CD, Perini A, Tonella P, Kessler FB. Automated continuous testing of multiagent systems. In:Proc. of the 5th European Workshop on Multi-Agent Systems (EUMAS). 2007.
    [38] Soetens QD, Demeyer S, Zaidman A. Change-Based test selection in the presence of developer tests. In:Proc. of the 17th IEEE European Conf. on Software Maintenance and Reengineering (CSMR). 2013. 101-110.
    [39] Vöst S, Wagner S. Trace-Based test selection to support continuous integration in the automotive industry. In:Proc. of the Int'l Workshop on Continuous Software Evolution and Delivery. ACM, 2016. 34-40.
    [40] Campos J, Arcuri A, Fraser G, Abreu R. Continuous test generation:Enhancing continuous integration with automated test generation. European Journal of Oral Sciences, 2010,109(4):241-248.
    [41] Marijan D. Multi-Perspective regression test prioritization for time-constrained environments. In:Proc. of the IEEE Int'l Conf. on Software Quality, Reliability and Security (QRS). 2015. 157-162.
    [42] Gligoric M, Eloussi L, Marinov D. Practical regression test selection with dynamic file dependencies. In:Proc. of the 2015 Int'l Symp. on Software Testing and Analysis. ACM, 2015. 211-222.
    [43] Legunsen O, Hariri F, Shi A, Yu YF, Zhang LM, Marinov D. An extensive study of static regression test selection in modern software evolution. In:Proc. of the ACM SIGSOFT Int'l Symp. on Foundations of Software Engineering. 2016. 583-594.
    [44] Mirarab S, Akhlaghi S, Tahvildari L. Size-Constrained regression test case selection using multicriteria optimization. IEEE Trans. on Software Engineering, 2012,38(4):936-956.
    [45] Vasic M, Parvez Z, Milicevic A, Gligoric M. File-Level vs. module-level regression test selection for.NET. In:Proc. of the Joint Meeting on the Foundations of Software Engineering. ACM, 2017. 848-853.
    [46] Jiang B, Zhang Z, Tse TH, Chen TY. How well do test case prioritization techniques support statistical fault localization. In:Proc. of the 33rd Annual IEEE Int'l Computer Software and Applications Conf. 2009,1:99-106.
    [47] Do H, Mirarab S, Tahvildari L, Rothermel G. The effects of time constraints on test case prioritization:A series of controlled experiments. IEEE Trans. on Software Engineering, 2010,36(5):593-617.
    [48] Knauss E, Staron M, Meding W, Söder O, Nilsson A, Castell M. Supporting continuous integration by code-churn based test selection. In:Proc. of the IEEE Int'l Workshop on Rapid Continuous Software Engineering. 2015. 19-25.
    [49] Dösinger S, Mordinyi R, Biffl S. Communicating continuous integration servers for increasing effectiveness of automated testing. In:Proc. of the 27th IEEE/ACM Int'l Conf. on Automated Software Engineering. 2012. 374-377.
    [50] Marijan D, Liaaen M. Effect of time window on the performance of continuous regression testing. In:Proc. of the IEEE Int'l Conf. on Software Maintenance and Evolution. 2016. 568-571.
    [51] Cho Y, Kim J, Lee E. History-Based test case prioritization for failure information. In:Proc. of the 23rd IEEE Asia-Pacific Software Engineering Conf. (APSEC). 2016. 385-388.
    [52] Marijan D, Liaaen M. Test prioritization with optimally balanced configuration coverage. In:Proc. of the IEEE Int'l Symp. on High Assurance Systems Engineering. IEEE, 2017. 100-103.
    [53] Yoo S, Nilsson R, Harman M. Faster fault finding at Google using multi objective regression test optimization. In:Proc. of the 8th European Software Engineering Conf. and the ACM SIGSOFT Symp. on the Foundations of Software Engineering (ESEC/FSE 2011). Szeged, 2011.
    [54] Marijan D, Gotlieb A, Sen S. Test case prioritization for continuous regression testing:An industrial case study. In:Proc. of the 29th IEEE Int'l Conf. on Software Maintenance (ICSM). 2013. 540-543.
    [55] Biswas S, Blanton RD. Reducing test execution cost of integrated, heterogeneous systems using continuous test data. IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems, 2011,30(1):148-158.
    [56] Celik A, Vasic M, Milicevic A, Gligoric M. Regression test selection across JVM boundaries. In:Proc. of the 11th Joint Meeting on Foundations of Software Engineering. 2017. 809-820.
    [57] Mondal D, Hemmati H, Durocher S. Exploring test suite diversification and code coverage in multi-objective test case selection. In:Proc. of the IEEE Int'l Conf. on Software Testing, Verification and Validation. 2015. 1-10.
    Cited by
    Comments
    Comments
    分享到微博
    Submit
Get Citation

李英玲,王青.持续集成测试用例集优化综述研究.软件学报,2018,29(10):3021-3050

Copy
Share
Article Metrics
  • Abstract:3322
  • PDF: 6665
  • HTML: 4589
  • Cited by: 0
History
  • Received:January 11,2018
  • Revised:May 01,2018
  • Online: October 12,2018
You are the first2036684Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address:4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code:100190
Phone:010-62562563 Fax:010-62562533 Email:jos@iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063