An extensive study of static regression test selection in modern software evolution

Legunsen, Owolabi; Hariri, Farah; Shi, August; Lu, Yafeng; Zhang, Lingming; Marinov, Darko

doi:10.1145/2950290.2950361

Cited by 100 publications

(111 citation statements)

References 46 publications

(95 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Researchers have proposed SRTS techniques that track dependencies at different granularity levels [Badri et al 2005;Kung et al 1995;Orso et al 2004;Ren et al 2003]. We performed an extensive prior study of SRTS techniques that track dependencies at both class and method levels [Legunsen et al 2016]. In that study, we selected to run tests at the level of test classes as opposed to test methods (selecting to run all test methods in an affected test class) 1 Experimental results from our prior study [Legunsen et al 2016] showed that method-level SRTS based on method call graphs is much more imprecise/unsafe and costly than class-level SRTS based on class-level dependencies.…”

Section: Static Regression Test Selectionmentioning

confidence: 99%

“…We measure the benefits and costs of these RA techniques relative to RetestAll, and to reflectionunaware static RTS [Legunsen et al 2016], henceforth called RU Analysis. We previously implemented RU Analysis in our open-source tool, STARTS [Legunsen et al 2017; STARTS Team 2018], and we build the RA techniques on the RU Analysis in STARTS.…”

Section: Introductionmentioning

confidence: 99%

“…Evaluating test dependencies provides more insights into (potential) test-selection behavior of RTS techniquesÐit can help understand why a technique selects or misses to select a test. No prior study of RTS [Yoo and Harman 2012], including our own recent studies [Legunsen et al 2016;Zhang 2018;Zhu et al 2019], evaluated the computed test dependencies. Evaluating test dependencies can help to understand whether static RTS was safe in prior studies in the presence of reflection because (1) test dependencies are not under-approximated by missing reflective edges, or (2) test dependencies are under-approximated, but actual code changes do not frequently touch dependencies that are only reachable via reflective edges.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Reflection-aware static regression test selection

Shi

Hadzi-Tanovic

Zhang³

et al. 2019

Proc. ACM Program. Lang.

Self Cite

View full text Add to dashboard Cite

Regression test selection (RTS) aims to speed up regression testing by rerunning only tests that are affected by code changes. RTS can be performed using static or dynamic analysis techniques. Our prior study showed that static and dynamic RTS perform similarly for medium-sized Java projects. However, the results of that prior study also showed that static RTS can be unsafe, missing to select tests that dynamic RTS selects, and that reflection was the only cause of unsafety observed among the evaluated projects. In this paper, we investigate five techniquesÐthree purely static techniques and two hybrid static-dynamic techniquesÐthat aim to make static RTS safe with respect to reflection. We implement these reflection-aware (RA) techniques by extending the reflection-unaware (RU) class-level static RTS technique in a tool called STARTS. To evaluate these RA techniques, we compare their end-to-end times with RU, and with RetestAll, which reruns all tests after every code change. We also compare safety and precision of the RA techniques with Ekstazi, a state-of-the-art dynamic RTS technique; precision is a measure of unaffected tests selected. Our evaluation on 1173 versions of 24 open-source Java projects shows negative results. The RA techniques improve the safety of RU but at very high costs. The purely static techniques are safe in our experiments but decrease the precision of RU, with end-to-end time at best 85.8% of RetestAll time, versus 69.1% for RU. One hybrid static-dynamic technique improves the safety of RU but at high cost, with end-to-end time that is 91.2% of RetestAll. The other hybrid static-dynamic technique provides better precision, is safer than RU, and incurs lower end-to-end timeÐ75.8% of RetestAll, but it can still be unsafe in the presence of test-order dependencies. Our study highlights the challenges involved in making static RTS safe with respect to reflection. CCS Concepts: • Software and its engineering → Automated static analysis; Software testing and debugging; Software evolution.

show abstract

Section: Static Regression Test Selectionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reflection-aware static regression test selection

Shi

Hadzi-Tanovic

Zhang³

et al. 2019

Proc. ACM Program. Lang.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Another approach by this author has been to dynamically detect, at the operating system level, all file artifacts a test depends on . Legunsen et al reported a study evaluating the performance of static regression test selection techniques, and the authors reported that class‐level approaches perform better than method‐level approaches. Zhang proposed a hybrid regression test selection approach combining method and file level selection .…”

Section: Related Workmentioning

confidence: 99%

A learning algorithm for optimizing continuous integration development and testing practice

2018

View full text Add to dashboard Cite

Summary Continuous integration, at its core, includes a set of practices that aim to prevent and reduce the cost of software integration issues by merging working software copies often. Regression testing is considered a good practice in software development with continuous integration, which ensures that code changes are not negatively affecting software functionality. As, nowadays, software development is carried out iteratively, with small code increments continuously developed and regression tested, it is of critical importance that continuous regression testing is time efficient. However, in practice, regression testing is often long lasting and faces scalability problems as software grows larger or as software changes are made more frequently. One contributing factor to these issues is test redundancy, which causes the same software functionality being tested multiple times across a test suite. In large‐scale software, especially highly configurable software, redundancy in continuous regression testing can significantly grow the size of test suites and negatively affect the cost effectiveness of continuous integration. This paper presents a practical learning algorithm for optimizing continuous integration testing by reducing ineffective test redundancy in regression suites. The novelty of the algorithm lies in learning and predicting the fault‐detection effectiveness of continuous integration tests using historical test records and combining this information with coverage‐based redundancy metrics. The goal is to identify ineffective redundancy, which is maximally reduced in the resulting regression test suite, thus reducing test time and improving the performance of continuous integration. We apply and evaluate the algorithm in two industrial projects of continuous integration. The results show that the proposed algorithm can improve the efficiency of continuous integration practice in terms of decreasing test execution time by 38% on average compared to the industry practice of our case study and by 40% on average compared to the retest‐all approach. The results further demonstrate no significant reduction in fault‐detection effectiveness of continuous regression testing. This suggests that the proposed algorithm contributes to the state of the practice in the continuous integration development and testing of highly configurable systems.

show abstract

“…While dynamic TCP techniques can be useful in practice, they may not be always applicable due to certain notable shortcomings, including: 1) the time cost of executing an instrumented program to collect coverage information [25], [50]; 2) expensive storage and maintenance of coverage information [50], [76]; 3) imprecise coverage metrics due to code changes during evolution or thread scheduling of concurrent systems [44], and 4) the absence of coverage information for newly added tests [46] or systems/modules that disallow code instrumentation [44] (e.g., code instrumentation may break the time constraints of real-time systems). Thus, to offer alternative solutions that that do not exhibit many of these shortcomings, researchers have proposed a number of TCP techniques that rely solely upon arXiv:1806.09774v1 [cs.SE] 26 Jun 2018 static information extracted from the text of source and test code.…”

Section: Introductionmentioning

confidence: 99%

How Do Static and Dynamic Test Case Prioritization Techniques Perform on Modern Software Systems? An Extensive Study on GitHub Projects

Luo

Moran

Zhang³

et al. 2019

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Test Case Prioritization (TCP) is an increasingly important regression testing technique for reordering test cases according to a pre-defined goal, particularly as agile practices gain adoption. To better understand these techniques, we perform the first extensive study aimed at empirically evaluating four static TCP techniques, comparing them with state-of-research dynamic TCP techniques across several quality metrics. This study was performed on 58 real-word Java programs encompassing 714 KLoC and results in several notable observations. First, our results across two effectiveness metrics (the Average Percentage of Faults Detected APFD and the cost cognizant APFDc) illustrate that at test-class granularity, these metrics tend to correlate, but this correlation does not hold at test-method granularity. Second, our analysis shows that static techniques can be surprisingly effective, particularly when measured by APFDc. Third, we found that TCP techniques tend to perform better on larger programs, but that program size does not affect comparative performance measures between techniques. Fourth, software evolution does not significantly impact comparative performance results between TCP techniques. Fifth, neither the number nor type of mutants utilized dramatically impact measures of TCP effectiveness under typical experimental settings. Finally, our similarity analysis illustrates that highly prioritized test cases tend to uncover dissimilar faults. and his thesis focused on improving bug reporting for mobile apps through novel applications of program analysis techniques. He has published in several top peerreviewed software engineering venues including: ICSE, ESEC/FSE, IC-SME, ICST, and MSR. He was recently recognized as the second-overall winner among graduate students in the ACM Student Research competition at ESEC/FSE15. Moran is a student member of IEEE and ACM and has served as an external reviewer for ICSE, ICSME, FSE, APSEC, and SCAM. More information available at https://www.kpmoran.com.

show abstract

An extensive study of static regression test selection in modern software evolution

Cited by 100 publications

References 46 publications

Reflection-aware static regression test selection

Reflection-aware static regression test selection

A learning algorithm for optimizing continuous integration development and testing practice

How Do Static and Dynamic Test Case Prioritization Techniques Perform on Modern Software Systems? An Extensive Study on GitHub Projects

Contact Info

Product

Resources

About