Towards Raising the Failure of Unit Tests to the Level of Compiler-Reported Errors

Abstract

Abstract. Running unit tests suites with contemporary tools such as JUNIT can show the presence of bugs, but not their locations. This is different from checking a program with a compiler, which always points the programmer to the most likely causes of the errors it detects. We argue that there is enough information in test suites and the programs under test to exclude many locations in the source as reasons for the failure of test cases, and further to rank the remaining locations according to derived evidence of their faultiness. We present a framework for the management of fault locators whose error diagnoses are based on data about a program and its test cases, especially as collected during test runs, and demonstrate that it is capable of performing reasonably well using a couple of simple fault locators in different evaluation scenarios.

A Simple Coverage-Based Locator for Multiple Faults

“…is the square of the Ochiai similarity coefficient [1], and has been shown independently in [1] and [16] to yield excellent results in presence of a single fault. However, given that all failed test cases must have executed the single faulty unit (or otherwise they would not have failed), and given that FA/Ochiai punishes units that are not called by all failed test cases [16], this should come as no surprise.…”

“…However, given that all failed test cases must have executed the single faulty unit (or otherwise they would not have failed), and given that FA/Ochiai punishes units that are not called by all failed test cases [16], this should come as no surprise.…”

“…[ [16]). The body of literature presenting new fault locators of either kind is continuously growing, with evaluations of their capability showing that presented fault locators excel previous ones, either generally or under certain conditions.…”

“…Elsewhere [16], we have divided fault localization methods into ones that exploit a priori hints at faultiness (such as program metrics; e.g. [ [16]).…”

“…A particular breed of fault locators, called hit spectra based or coverage based fault locators [1] [6][7] [11] [ 16], exploits information gathered from passed and failed test cases. The general idea behind this is that program units under test (hereafter referred to as UUTs) executed by more failed and fewer passed test cases are more likely faulty than others.…”