Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings

Abstract

Abstract-Software defect prediction strives to improve software quality and testing efficiency by constructing predictive classification models from code attributes to enable a timely identification of fault-prone modules. Several classification models have been evaluated for this task. However, due to inconsistent findings regarding the superiority of one classifier over another and the usefulness of metric-based classification in general, more research is needed to improve convergence across studies and further advance confidence in experimental results. We consider three potential sources for bias: comparing classifiers over one or a small number of proprietary data sets, relying on accuracy indicators that are conceptually inappropriate for software defect prediction and cross-study comparisons, and, finally, limited use of statistical testing procedures to secure empirical findings. To remedy these problems, a framework for comparative software defect prediction experiments is proposed and applied in a large-scale empirical comparison of 22 classifiers over 10 public domain data sets from the NASA Metrics Data repository. Overall, an appealing degree of predictive accuracy is observed, which supports the view that metric-based classification is useful. However, our results indicate that the importance of the particular classification algorithm may be less than previously assumed since no significant performance differences could be detected among the top 17 classifiers.

A Hierarchical Feature Set optimization for effective code change based Defect Forecasting

“…Information has been also used by earlier studies ( [16] NNP [17] [18] NNS NASA classifier. SVM MDP repository and an.…”

A Deep Analysis of the Precision Formula for Imbalanced Class Distribution

“…3. In the domain of software defect prediction, often the data sets under study represents less than 1 percent of the data point in total [1], [10] and [11]. Referencing [5] presented an example of using the most imbalanced of the NASA Metric Data Program; PC2.…”

Influence of confirmation biases of developers on software quality: an empirical study

“…In software defect prediction studies, it is also empirically shown that the performance of the Naïve Bayes is amongst the top algorithms [17]. As shown in Table 6, datasets are imbalanced.…”

“…As a result, both increasing the efficiency of the software testing phase and delivering the software product to the market on time become possible. Reported results in software defect prediction literature suggest that further progress in defect prediction performance can be achieved by increasing the content of input data that defect predictors learn rather than using different algorithms or increasing the size of input data [17], [15], [16]. We can group some significant work in the literature in terms of their focus: algorithm driven approaches; data size driven approaches; and data content driven approaches.…”