An Empirical Study of JUnit Test-Suite Reduction

Zhang, Lingming; Marinov, Darko; Lu, Zixing; Khurshid, Sarfraz

doi:10.1109/issre.2011.26

Cited by 57 publications

(44 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To compare VmVm to TSM, we turned to a study performed by Zhang et al [18], which evaluated the performance of four TSM techniques. Zhang et al studied the performance of each technique in terms of fault finding reduction and test suite size reduction across several version of four real-world Java applications downloaded from the Software Infrastructure Repository [7].…”

Section: Discussionmentioning

confidence: 99%

“…We downloaded the same applications, replicating their experiments using VmVm instead of the TSM mechanisms from their study, comparing the results. VmVm showed a reduction in test suite execution time of 42% with no loss of fault finding ability, while the TSM technique we compared to (there were several studied by [18]; we selected the most stable in terms of fault finding ability) showed a reduction in test suite size of 12%. From these results (described further in [3]), we conclude that VmVm can provide a significant reduction in test suite execution time, compared to existing TSM mechanisms.…”

Section: Discussionmentioning

confidence: 99%

“…JTor, an alpha-quality Tor implementation with only 445 commits and 6 contributors overall). On average, each application had 475,000 lines of code, 56 test classes, and was 7.32 years old (significantly larger than the applications used to compare to TSM and used by [18]). For all twenty applications tested, VmVm was able to successfully instrument the application and isolate the test cases effectively, providing an average speedup of 62% with a maximum speedup of 97% (compared to isolating each test case in its own process).…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

VMVM: unit test virtualization for Java

Bell

Kaiser

2014

Companion Proceedings of the 36th International Conference on Software Engineering

View full text Add to dashboard Cite

As software evolves and grows, its regression test suites tend to grow as well. When these test suites become too large, they can eventually reach a point where they become too length to regularly execute. Previous work in Test Suite Minimization has reduced the number of tests in such suites by attempting to identify those that are redundant (e.g. by a coverage metric). Our approach to ameliorating the runtime of these large test suites is complementary, instead focusing on reducing the overhead of running each test, an approach that we call Unit Test Virtualization. This Tool Demonstration presents our implementation of Unit Test Virtualization, VmVm (pronounced "vroom-vroom") and summarizes an evaluation of our implementation on 20 real-world Java applications, showing that it reduces test suite execution time by up to 97% (on average, 62%). A companion video to this demonstration is available online, at https://www.youtube.com/watch?v=sRpqF3rJERI.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

VMVM: unit test virtualization for Java

Bell

Kaiser

2014

Companion Proceedings of the 36th International Conference on Software Engineering

View full text Add to dashboard Cite

show abstract

“…After a program is modified, developers often reuse existing tests for the program before modification and may add some tests for the modification. As it is time-consuming to run the aggregated tests, many test selection and/or reduction techniques [5,21,65,71] have been proposed to reduce the number of tests used in regression testing. To optimize the cost spent on regression testing, test prioritization techniques [61,62,68] have been proposed to schedule the execution order of tests.…”

Section: Regression Testingmentioning

confidence: 99%

Is This a Bug or an Obsolete Test?

Hao

Lan

Zhang

et al. 2013

ECOOP 2013 – Object-Oriented Programming

Self Cite

View full text Add to dashboard Cite

Abstract. In software evolution, developers typically need to identify whether the failure of a test is due to a bug in the source code under test or the obsoleteness of the test code when they execute a test suite. Only after finding the cause of a failure can developers determine whether to fix the bug or repair the obsolete test. Researchers have proposed several techniques to automate test repair. However, test-repair techniques typically assume that test failures are always due to obsolete tests. Thus, such techniques may not be applicable in real world software evolution when developers do not know whether the failure is due to a bug or an obsolete test. To know whether the cause of a test failure lies in the source code under test or in the test code, we view this problem as a classification problem and propose an automatic approach based on machine learning. Specifically, we target Java software using the JUnit testing framework and collect a set of features that may be related to failures of tests. Using this set of features, we adopt the Best-first Decision Tree Learning algorithm to train a classifier with some existing regression test failures as training instances. Then, we use the classifier to classify future failed tests. Furthermore, we evaluated our approach using two Java programs in three scenarios (within the same version, within different versions of a program, and between different programs), and found that our approach can effectively classify the causes of failed tests.

show abstract

“…Test-suite reduction [22][23][24]29,31,38,41,42] is an approach to make regression testing faster. The goal is to identify tests that can be removed from a test suite without substantially reducing its fault-detection capability.…”

Section: Introductionmentioning

confidence: 99%

Balancing trade-offs in test-suite reduction

Shi

Gyori

Gligoric

et al. 2014

Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering

Self Cite

View full text Add to dashboard Cite

Regression testing is an important activity but can get expensive for large test suites. Test-suite reduction speeds up regression testing by identifying and removing redundant tests based on a given set of requirements. Traditional research on test-suite reduction is rather diverse but most commonly shares three properties: (1) requirements are defined by a coverage criterion such as statement coverage;(2) the reduced test suite has to satisfy all the requirements as the original test suite; and (3) the quality of the reduced test suites is measured on the software version on which the reduction is performed. These properties make it hard for test engineers to decide how to use reduced test suites.We address all three properties of traditional test-suite reduction: (1) we evaluate test-suite reduction with requirements defined by killed mutants; (2) we evaluate inadequate reduction that does not require reduced test suites to satisfy all the requirements; and (3) we propose evolution-aware metrics that evaluate the quality of the reduced test suites across multiple software versions. Our evaluations allow a more thorough exploration of trade-offs in test-suite reduction, and our evolution-aware metrics show how the quality of reduced test suites can change after the version where the reduction is performed. We compare the trade-offs among various reductions on 18 projects with a total of 261,235 tests over 3,590 commits and a cumulative history spanning 35 years of development. Our results help test engineers make a more informed decision about balancing size, coverage, and fault-detection loss of reduced test suites.

show abstract

An Empirical Study of JUnit Test-Suite Reduction

Cited by 57 publications

References 29 publications

VMVM: unit test virtualization for Java

VMVM: unit test virtualization for Java

Is This a Bug or an Obsolete Test?

Balancing trade-offs in test-suite reduction

Contact Info

Product

Resources

About