An Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application

Almasi, M.; Hemmati, Hadi; Fraser, Gordon; Arcuri, Andrea; Benefelds, Janis

doi:10.1109/icse-seip.2017.27

Cited by 121 publications

(81 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…There are 7 cases with high mutation score for which DSpot does not improve the number of killed mutants. In 5 of these cases, the original mutation score is greater than 87% (IDs: 2,7,12,21,22), and DSpot does not manage to synthesize improved inputs to cover new mutants and eventually kill them. In some cases DSpot cannot improve the test class because they rely on an external resource (a jar file), or use utility methods that are not considered as test methods by DSpot and hence are not modified by our tool.…”

Section: Answer To Rq3mentioning

confidence: 99%

See 1 more Smart Citation

Automatic test improvement with DSpot: a study with ten mature open-source projects

et al. 2019

View full text Add to dashboard Cite

In the literature, there is a rather clear segregation between manually written tests by developers and automatically generated ones. In this paper, we explore a third solution: to automatically improve existing test cases written by developers. We present the concept, design and implementation of a system called DSpot, that takes developer-written test cases as input (JUnit tests in Java) and synthesizes improved versions of them as output. Those test improvements are given back to developers as patches or pull requests, that can be directly integrated in the main branch of the test code base. We have evaluated DSpot in a deep, systematic manner over 40 real-world unit test classes from 10 notable and open-source software projects. We have amplified all test methods from those 40 unit test classes. In 26/40 cases, DSpot is able to 4 Benjamin Danglot et al. DSpotDSpot is an automatic test improvement tool for Java unit tests. It is built upon the algorithms of Tonella [25] and Xie [28]. DSpot inputsThe input of DSpot consists in a set of existing test cases, manually written by the developers. As output, DSpot produces variants of the given test cases. These variants are meant to be added to the test suite. By putting together existing test cases and their variants, we aim at strictly improving the overall test suite quality. By construction, the enhanced test suite is at least as good, or better than the original one w.r.t. the considered criterion.Concretely, DSpot synthesizes suggestions in the form of diffs that are proposed to the developer: Figure 1 shows such a test improvement.

show abstract

Section: Answer To Rq3mentioning

confidence: 99%

“…Almasi et al [2] investigate the efficiency and effectiveness of automated test generation on a production ready application named LifeCalc. They use 25 real faults from LifeCalc to evaluate two state-of-the-art tools, Evosuite and Randoop, by asking feedback from the developers about the generated test methods.…”

Section: Related Workmentioning

confidence: 99%

Automatic test improvement with DSpot: a study with ten mature open-source projects

et al. 2019

View full text Add to dashboard Cite

show abstract

“…To our knowledge, EvoSuite is the state-of-art open source Java unit test generation tool. Compared with another popular test generation tool Randoop (Pacheco and Ernst (2007)), some recent studies (Almasi et al (2017); Shamshiri et al (2015)) have shown that Evosuite is better than Randoop in terms of a) compilable test generated, b) minimized flakiness, c) false positives, d) coverage, and e) most importantly-the number of bugs detected. While the generated tests by EvoSuite can possibly have problems of creating complex objects, exposing complex conditions, accessing private methods or fields, creating complex interactions, and generating appropriate assertions, they can be considered as effective in finding bugs in open-source and industrial systems in general (Shamshiri et al (2015)).…”

Section: Automatic Test Case Generation Toolmentioning

confidence: 99%

Alleviating patch overfitting with automatic test generation: a study of feasibility and effectiveness for the Nopol repair system

Martínez

Danglot

et al. 2018

Empir Software Eng

View full text Add to dashboard Cite

Among the many different kinds of program repair techniques, one widely studied family of techniques is called test suite based repair. However, test suites are in essence input-output specifications and are thus typically inadequate for completely specifying the expected behavior of the program under repair. Consequently, the patches generated by test suite based repair techniques can just overfit to the used test suite, and fail to generalize to other tests. We deeply analyze the overfitting problem in program repair and give a classification of this problem. This classification will help the community to better understand and design techniques to defeat the overfitting problem. We further propose and evaluate an approach called UnsatGuided, which aims to alleviate the overfitting problem for synthesis-based repair techniques with automatic test case generation. The approach uses additional automatically generated tests to strengthen the repair constraint used by synthesis-based repair techniques. We analyze the effectiveness of UnsatGuided: 1) analytically with respect to alleviating two different kinds of overfitting issues; 2) empirically based on an experiment over the 224 bugs of the Defects4J repository. The main result is that automatic test generation is effective in alleviating one kind of overfitting issue-regression introduction, but due to oracle problem, has minimal positive impact on alleviating the other kind of overfitting issue-incomplete fixing.

show abstract

“…It automatically generates test cases using assertions. The performance of this tool is further demonstrated and compared in other prominent studies [51][52][53][54][55].…”

mentioning

confidence: 96%

“…It automatically generates test cases using assertions. The performance of this tool is further demonstrated and compared in other prominent studies [51][52][53][54][55].Considering the object's state, Bashir and Nadeem [56] proposed a novel fitness function, which restricts the search process by reviewing the tests that have either obtained the desired state or requires more method calls. The authors in [57][58][59] further extended the work that resulted in the development of a tool named eMuJava.…”

mentioning

confidence: 99%

On the Effectiveness of Using Elitist Genetic Algorithm in Mutation Testing

2019

View full text Add to dashboard Cite

Manual test case generation is an exhaustive and time-consuming process. However, automated test data generation may reduce the efforts and assist in creating an adequate test suite embracing predefined goals. The quality of a test suite depends on its fault-finding behavior. Mutants have been widely accepted for simulating the artificial faults that behave similarly to realistic ones for test data generation. In prior studies, the use of search-based techniques has been extensively reported to enhance the quality of test suites. Symmetry, however, can have a detrimental impact on the dynamics of a search-based algorithm, whose performance strongly depends on breaking the “symmetry” of search space by the evolving population. This study implements an elitist Genetic Algorithm (GA) with an improved fitness function to expose maximum faults while also minimizing the cost of testing by generating less complex and asymmetric test cases. It uses the selective mutation strategy to create low-cost artificial faults that result in a lesser number of redundant and equivalent mutants. For evolution, reproduction operator selection is repeatedly guided by the traces of test execution and mutant detection that decides whether to diversify or intensify the previous population of test cases. An iterative elimination of redundant test cases further minimizes the size of the test suite. This study uses 14 Java programs of significant sizes to validate the efficacy of the proposed approach in comparison to Initial Random tests and a widely used evolutionary framework in academia, namely Evosuite. Empirically, our approach is found to be more stable with significant improvement in the test case efficiency of the optimized test suite.

show abstract

An Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application

Cited by 121 publications

References 20 publications

Automatic test improvement with DSpot: a study with ten mature open-source projects

Automatic test improvement with DSpot: a study with ten mature open-source projects

Alleviating patch overfitting with automatic test generation: a study of feasibility and effectiveness for the Nopol repair system

On the Effectiveness of Using Elitist Genetic Algorithm in Mutation Testing

Contact Info

Product

Resources

About