Improving test suites for efficient fault localization

Abstract

The need for testing-for-diagnosis strategies has been identified for a long time, but the explicit link from testing to diagnosis (fault localization) is rare. Analyzing the type of information needed for efficient fault localization, we identify the attribute (called Dynamic Basic Block) that restricts the accuracy of a diagnosis algorithm. Based on this attribute, a test-for-diagnosis criterion is proposed and validated through rigorous case studies: it shows that a test suite can be improved to reach a high level of diagnosis accuracy. So, the dilemma between a reduced testing effort (with as few test cases as possible) and the diagnosis accuracy (that needs as much test cases as possible to get more information) is partly solved by selecting test cases that are dedicated to diagnosis.

Effective Fault Localization Approach Using Feedback

“…SFL is independent of any specific model of system and incurs low time and space overhead. Due to the feature of statistics and its simplicity, SFL is widely accepted and studied as a promising technique in the fault localization community [2]- [5], [8]- [21], and the research [2], [3] has also empirically proven that SFL has high effectiveness of locating faults. Therefore, our approach is applied to SFL to fully demonstrate its effectiveness.…”

“…The performance of SFL is widely evaluated by the percentage of code that needs to be examined (or not examined) to find the fault [2]- [5], [8], [9], [11]- [15], [17]- [21]. This evaluation assumes that debugging engineers will examine all statements from top to bottom according to the ranking list given by the SFL until they encounter the faulty statement.…”

“…However, most fault localization approaches usually ignore this interaction [8]- [21]. They assume that test cases satisfying a certain test adequacy criterion can provide adequate information for fault localization, and there are no new test cases to provide required feedback [5]. In addition, the goal of testing is to reveal faults instead of locating faults.…”

“…Typically, SFL exploits the correlations between program entities and program failures by statistically analyzing coverage information. SFL has been widely studied and used in the field of fault localization [2]- [5], [8]- [21], and the prior research [2], [3] has empirically proven that SFL yields a promising ability in locating faults.…”

Is non-parametric hypothesis testing model robust for statistical fault localization?

“…Each program is a variation of one of seven programs, namely "tcas", "tot info", "replace", "print tokens", "print tokens2", "schedule", and "schedule2", varying in sizes from 133 to 515 executable lines. 2 Each faulty version is seeded with one fault. These programs and their faulty versions are downloaded from the Software-artifact Infrastructure Repository (SIR) [5] website.…”