Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T)

Abstract

Rather than tediously writing unit tests manually, tools can be used to generate them automatically-sometimes even resulting in higher code coverage than manual testing. But how good are these tests at actually finding faults? To answer this question, we applied three state-of-the-art unit test generation tools for Java (Randoop, EvoSuite, and Agitar) to the 357 real faults in the Defects4J dataset and investigated how well the generated test suites perform at detecting these faults. Although the automatically generated test suites detected 55.7% of the faults overall, only 19.9% of all the individual test suites detected a fault. By studying the effectiveness and problems of the individual tools and the tests they generate, we derive insights to support the development of automated unit test generators that achieve a higher fault detection rate. These insights include 1) improving the obtained code coverage so that faulty statements are executed in the first instance, 2) improving the propagation of faulty program states to an observable output, coupled with the generation of more sensitive assertions, and 3) improving the simulation of the execution environment to detect faults that are dependent on external factors such as date and time.

Ensuring the Observability of Structural Test Obligations

“…Results to date are promising, given the complexity of some of the faults detected [58,2,21,3]. However, automated generation does not yet produce human competitive results [17].…”

Optimized Order of Software Testing Techniques in Agile Process – A Systematic Approach

“…Not necessarily all the components and functional areas must be considered for automated testing, rather it's the job of a test manager to decide which parts of the product should be considered for automated testing and which for manual or other testing strategies. The code coverage is measured through automated testing tools, however the effectiveness of faults detection on the basis of scripted unit tests has been demonstrated in [9].…”

Classification of Randomly Generated Test Cases

“…In their follow-up work, they analyze three automated test generators using Defects4j: Randoop [11], Evosuite [6], and Agitar [1], to study whether these automated tools are helpful to uncover real faults and the challenges of existing tools that limit that bug detection ability [14].…”