Evaluating defect prediction approaches: a benchmark and an extensive comparison

Abstract

Reliably predicting software defects is one of the holy grails of software engineering. Researchers have devised and implemented a plethora of defect/bug prediction approaches varying in terms of accuracy, complexity and the input data they require. However, the absence of an established benchmark makes it hard, if not impossible, to compare approaches. We present a benchmark for defect prediction, in the form of a publicly available dataset consisting of several software systems, and provide an extensive comparison of well-known bug prediction approaches, together with novel approaches we devised. We evaluate the performance of the approaches using different performance indicators: classification of entities as defect-prone or not, ranking of the entities, with and without taking into account the effort to review an entity. We performed three sets of experiments aimed at (1) comparing the approaches across different systems, (2) testing whether the differences in performance are statistically significant, and (3) investigating the stability of approaches across different learners. Our results indicate that, while some approaches perform better than others in a statistically significant manner, external validity in defect prediction is still an open problem, as generalizing results to different contexts/learners proved to be a partially unsuccessful endeavor.

An empirical study on predicting defect numbers

“…Surprisingly, the results about precision in Table III are, on average, higher than those in Table II, implying that CPDP achieves better performance than WPDP in this example. This finding is different from the results of many prior studies [3,9]. Similarly, Figure 3 shows that in the scenario of CPDP DTR is also the best estimator when considering precision.…”

“…Interestingly, several recent studies with respect to software defect classification [3,9,10] have found that simple classifiers, e.g., Naïve Bayes and Logistic Regression, were able to perform well in both within-project and cross-project scenarios, though those complex ones always achieved high precision. As we know, newly created or unpopular software projects have little historical data available to train any classifiers, which is very similar to the typical problem cold start in recommender systems [11].…”

Empirical Evaluation of Machine Learning Algorithms for Fault Prediction

“…The goal of our research is to empirically study performance of various classifiers for fault prediction on the data sets provided by Marco D'Ambros [45].…”

On software engineering repositories and their open problems

“…http://sourcerer.ics.uci.edu/ • Ultimate Debian Database (UDD) [11] http://udd.debian.org/ • Bug Prediction Dataset (BPD) [12], [13]:…”