Evaluating Defect Prediction Models for a Large Evolving Software System

Abstract

A plethora of defect prediction models has been proposed and empirically evaluated, often using standard classification performance measures.In this paper, we explore defect prediction models for a large, multi-release software system from the telecommunications domain. A history of roughly 3 years is analyzed to extract process and static code metrics that are used to build several defect prediction models with Random Forests.The performance of the resulting models is comparable to previously published work. Furthermore, we develop a new evaluation measure based on the comparison to an optimal model.

Revisiting the evaluation of defect prediction models

“…It seems that the trivial model performs better for data sets where the 90th percentile of LoC for fault-free files is lower. This can explain an observation we made in a recent study: We found out that different prediction models consistently performed better when header files where included [20]. Since header files are usually much shorter than their corresponding implementation files, and at least in our case, header files contained fewer defects, the 90th percentile for fault-free files was lower when we included header files.…”

“…They propose to use a variation of lift charts where the x-axis contains the ratio of lines of code instead of modules, and evaluate various data mining algorithms by measuring the performance improvement provided by a classification technique over a random selection of modules. In a recent paper, we have extended this approach to take the characteristics of the data set into account by measuring the deviation from an optimal model [20]. The resulting performance measure is described in Section 3.…”

“…It reports how much better than a random model a predictor is, but it does not tell anything about how close the predictor comes to an optimal model. Therefore we extended the idea of CE and proposed a new performance evaluation measure popt by comparing a prediction model with an optimal model [20]. An optimal model would be created as follows: We order all modules by decreasing defect density (and increasing lines of code, in case of ties), and predict the modules with highest defect densities first.…”

“…In a recent paper, we proposed a performance measure that takes these considerations into account and assesses defect prediction models by comparing them to the optimal performance of an imaginary best classifier [20]. However, this metric has been empirically validated only on confidential industrial data but not on publicly available data sets yet.…”

On the Utility of a Defect Prediction Model during HW/SW Integration Testing: A Retrospective Case Study

“…The evaluation of defect prediction models is a controversial topic, as recent studies show [7], [8], [20]- [22]. One problem is the choice of adequate performance measures, where adequate depends on usage scenarios, criticality, and budget.…”