Theory and Practice, Do They Match? A Case with Spectrum-Based Fault Localization

Abstract

Abstract-Spectrum-based fault localization refers to the process of identifying program units that are buggy from two sets of execution traces: normal traces and faulty traces. These approaches use statistical formulas to measure the suspiciousness of program units based on the execution traces. There have been many spectrum-based fault localization approaches proposing various formulas in the literature. Two of the best performing and well-known ones are Tarantula and Ochiai. Recently, Xie et al. [18] find that theoretically, under certain assumptions, two families of spectrum-based fault localization formulas outperform all other formulas including those of Tarantula and Ochiai. In this work, we empirically validate Xie et al.'s findings by comparing the performance of the theoretically best formulas against popular approaches on a dataset containing 199 buggy versions of 10 programs. Our empirical study finds that Ochiai and Tarantula statistically significantly outperforms 3 out of 5 theoretically best fault localization techniques. For the remaining two, Ochiai also outperforms them, albeit not statistically significantly. This happens because an assumption in Xie et al.'s work is not satisfied in many fault localization settings.

First, Debug the Test Oracle

“…This is in accordance with empirical study [2] as Ochiai outperforms Tarantula consistently. It also partially agrees with [15] as Ochiai performs superior than Naish2, but inferior than Rus-sel&Rao. This is because [15] compares the average accuracy of all program variants while we plotted the proportion of program variants with accuracy score less than s x when s x walks through [0, 1].…”

“…It also partially agrees with [15] as Ochiai performs superior than Naish2, but inferior than Rus-sel&Rao. This is because [15] compares the average accuracy of all program variants while we plotted the proportion of program variants with accuracy score less than s x when s x walks through [0, 1]. Results from both us and former empirical studies contradict the conclusion in [32] as Naish2 should be better than Ochiai theoretically.…”

“…Results from both us and former empirical studies contradict the conclusion in [32] as Naish2 should be better than Ochiai theoretically. In addition to the unrealistic assumption of 100% coverage pointed out by [15], we believe there are another two reasons causing this disagreement. Firstly proofs in [32] are based on "single fault assumption".…”

“…Therefore ER1 and ER5 were stated to be metrics with "maximal" accuracy. However results of empirical study in [15] states that Ochiai is statistically better than ER5 and ER1, and Tarantula's score is better than Naish1 and ER5. Russel&Rao a ef a ef +a nf +aep+anp They owe this contradiction to the unrealistic 100% coverage assumption in [32].…”

“…They further proved theoretically the "better than" relation among these groups and identified group ER1 (Naish1, Naish2) and ER5 (Binary, Russel&Rao, Wong1) to be "maximal" metrics under single-fault condition. [15] empirically studied the accuracy of Tarantula, Ochiai and all metrics in ER1 and ER5. Results show that Ochiai is better than ER1 and ER5, and Tarantula achieved better average accuracy than Naish1 and ER5.…”