Entropy-based test generation for improved fault localization

Abstract

Spectrum-based Bayesian reasoning can effectively rank candidate fault locations based on passing/failing test cases, but the diagnostic quality highly depends on the size and diversity of the underlying test suite. As test suites in practice often do not exhibit the necessary properties, we present a technique to extend existing test suites with new test cases that optimize the diagnostic quality. We apply probability theory concepts to guide test case generation using entropy, such that the amount of uncertainty in the diagnostic ranking is minimized. Our ENTBUG prototype extends the search-based test generation tool EVOSUITE to use entropy in the fitness function of its underlying genetic algorithm, and we applied it to seven real faults. Empirical results show that our approach reduces the entropy of the diagnostic ranking by 49% on average (compared to using the original test suite), leading to a 91% average reduction of diagnosis candidates needed to inspect to find the true faulty one.

Improving fault localization for Simulink models using search-based testing and prediction models

“…Coverage dissimilarity has previously been used for test prioritization [21], and is used in this paper for the first time to improve fault localization. The two other alternatives, i.e., coverage density [15] and number of dynamic basic blocks [16], have been previously used to improve source code fault localization. Coverage Dissimilarity.…”

“…Since statistical debugging is essentially heuristic, despite various research advancements, it still remains largely unpredictable [15]. In practice, it is likely that several elements have the same suspiciousness score as that of the faulty, and hence, be assigned the same rank.…”

“…Given the way statistical debugging works, if every test case in the test suite used for debugging executes either both or neither of a pair of elements, then those elements will have the same suspiciousness scores (i.e., they will be put in the same rank group). One promising strategy to improve precision of statistical debugging is to use an existing ranking to generate additional test cases that help refine the ranking by reducing the size of rank groups in the ranking [15]- [18].…”

“…This is because Simulink models include computationally expensive physical models [20], and their outputs are complex continuous signals [19]. We identify three alternative test objectives that aim to generate test cases exercising diverse parts of the underlying code and adapt these objectives to Simulink models [15], [16], [21]. We use these objectives to develop a search-based test generation algorithm, which builds on the whole test suite generation algorithm [22], to extend an existing test suite with a small number of test cases.…”

“…Having these rules and having a pair of statistical rankings from before and after adding some test cases, we can predict whether test generation should be stopped or continued. Our Contributions include: -We develop a search-based testing technique for Simulink models that uses the existing alternative test objectives [15], [16], [21] to generate small and diverse test suites that can help improve fault localization accuracy.…”