Using automated program repair for evaluating the effectiveness of fault localization techniques

Abstract

Many techniques on automated fault localization (AFL) have been introduced to assist developers in debugging. Prior studies evaluate the localization technique from the viewpoint of developers: measuring how many benefits that developers can obtain from the localization technique used when debugging. However, these evaluation approaches are not always suitable, because it is difficult to quantify precisely the benefits due to the complex debugging behaviors of developers. In addition, recent user studies have presented that developers working with AFL do not correct the defects more efficiently than ones working with only traditional debugging techniques such as breakpoints, even when the effectiveness of AFL is artificially improved.In this paper we attempt to propose a new research direction of developing AFL techniques from the viewpoint of fully automated debugging including the program repair of automation, for which the activity of AFL is necessary. We also introduce the NCP score as the evaluation measurement to assess and compare various techniques from this perspective. Our experiment on 15 popular AFL techniques with 11 subject programs shipping with real-life field failures presents the evidence that these AFL techniques performing well in prior studies do not have better localization effectiveness according to NCP score. We also observe that Jaccard has the better performance over other techniques in our experiment.

Ask the Mutants: Mutating Faulty Programs for Fault Localization

“…In addition, MUSE ranks the faulty statement at the top of the suspiciousness ranking for seven out of 14 studied faults, and within the top three for another three faults. In addition, the newly introduced LIL metric also shows that MUSE can be highly accurate, as well as confirming the observation made by Qi et al [23].…”

“…In an attempt to evaluate the precision of SBFL techniques, Qi et al [23] compared SBFL techniques by measuring the Number of Candidate Patches (NCP) generated by GenProg [25] automated program repair tool, with the given localization information.…”

“…An interesting empirical observation by Qi et al [23] is that Jaccard [10] produced lower NCP than Op2 [19], despite having been proven to always produce a lower ranking for the faulty statement than Op2 [28]. This is due to the actual distribution of the suspiciousness score: while Op2 produced higher ranking for the faulty statement than Jaccard, it assigned almost equally high suspiciousness scores to some correct statements.…”

“…RQ2 evaluates MUSE with the Expense metric against the three widely studied SBFL techniquesJaccard, Ochiai, and Op2. Op2 [19] is proven to perform well in Expense metric; Ochiai [20] performs closely to Op2, while Jaccard [10] shows good performance when used with automated program repair [23].…”

“…The traditional evaluation metric in SBFL literature is the Expense metric, which is the percentage of program statements the human developer needs to inspect before encountering the faulty one [18]. However, recent work showed that the Expense metric failed to account for the performance of the automated program repair tool that used various SBFL techniques to locate the fix: techniques proven to rank the faulty statement higher than others actually performed poorer when used in conjunction with a repair tool [23].…”