An Evaluation of Similarity Coefficients for Software Fault Localization

Abstract

Automated diagnosis of software faults can improve the efficiency of the debugging process, and is therefore an important technique for the development of dependable software. In this paper we study different similarity coefficients that are applied in the context of a program spectral approach to software fault localization (single programming mistakes). The coefficients studied are taken from the systems diagnosis / automated debugging tools Pinpoint, Tarantula, and AMPLE, and from the molecular biology domain (the Ochiai coefficient). We evaluate these coefficients on the Siemens Suite of benchmark faults, and assess their effectiveness in terms of the position of the actual fault in the probability ranking of fault candidates produced by the diagnosis technique. Our experiments indicate that the Ochiai coefficient consistently outperforms the coefficients currently used by the tools mentioned. In terms of the amount of code that needs to be inspected, this coefficient improves 5% on average over the next best technique, and up to 30% in specific cases.

Fault localization in embedded software based on a single cyclic trace

“…• Tarantula's, the original coefficient used in [8] to assist fault localization using a visualization technique; • The Jaccard index, a well-known statistic measure used to compare sets and used in the Pinpoint framework [24]; • AMPLE, the coefficient used in the tool AMPLE (Analyzing Method Patterns to Locate Errors) to locate error in object-oriented software [25]; • Ochiai, a coefficient originally used in molecular biology, but used successfully in fault localization [1];…”

“…Then, it computes statements that are executed by one but not the other, using the so-called union and intersection models. Other methods compares executions statistically [8], [1], [9], [11]. In particular, the Tarantula approach [8] computes a measure of association between executions of program statements and failures.…”

“…This approach is based on Automatic Fault-Localization (AFL) methods [7], [8], [1], [9] and takes advantage of the cyclic nature of traces. Our approach first automatically detects the cycles in the trace before using adapted AFL method to find the most suspicious statements.…”

“…In this section, we focus on two existing fault-localization methods: the nearest neighbor method [7] and the statistical method [8], [1]. Our approach is based on these two methods.…”

Analysis of service diagnosis improvement through increased monitoring granularity

“…The similarity denotes the likelihood of a component being the faulty one and, therefore, determines its position in the diagnosis ranking. Any SC may be used; however, the Ochiai SC has been found to work best (Abreu et al 2006). Intuitively, SFL works by comparing the different combinations of component involvements in the individual system operations.…”