Is "better data" better than "better data miners"?

Abstract

We report and fix an important systematic error in prior studies that ranked classifiers for software analytics. Those studies did not (a) assess classifiers on multiple criteria and they did not (b) study how variations in the data affect the results. Hence, this paper applies (a) multi-performance criteria while (b) fixing the weaker regions of the training data (using SMOTUNED, which is an autotuning version of SMOTE). This approach leads to dramatically large increases in software defect predictions when applied in a 5*5 cross-validation study for 3,681 JAVA classes (containing over a million lines of code) from open source systems, SMOTUNED increased AUC and recall by 60% and 20% respectively. These improvements are independent of the classifier used to predict for defects. Same kind of pattern (improvement) was observed when a comparative analysis of SMOTE and SMOTUNED was done against the most recent class imbalance technique.In conclusion, for software analytic tasks like defect prediction, (1) data pre-processing can be more important than classifier choice, (2) ranking studies are incomplete without such pre-processing, and (3) SMOTUNED is a promising candidate for pre-processing.• RQ1: Are the default "off-the-shelf" parameters for SMOTE appropriate for all data sets?Result 1 SMOTUNED learned different parameters for each data set, all of which were very different from default SMOTE.• RQ2: Is there any benefit in tuning the default parameters of SMOTE for each new data set? Result 2Performance improvements using SMOTUNED are dramatically large, e.g., improvements in AUC up to 60% against SMOTE.In those results, we see that while no learner was best across all data sets and performance criteria, SMOTUNED was most often seen in the best results. That is, creating better training data might be more important than the subsequent choice of classifiers.

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

“…EPV (Event-Per-Variables)-a measure to evaluate the risk of overfitting-is the ratio of events to the number of independent variables used to train a model. Formally, EPV = #events (e.g., #defective modules) #variables (1) where the event is the number of occurrences of the least frequently occurring class of the dependent variable (e.g., the numbers of defective modules), and the variables is the number of independent variables used to train the model (i.e., the number of software metrics) [85]. Recently, Tantithamthavorn et al [85] demonstrated that models that are trained using datasets where the EPV is low (i.e., too few events are available relative to the number of independent variables) are especially susceptible to overfitting (i.e., being fit too closely to the training data).…”

Multitask defect prediction

“…To minimize the threats, we use the F1 measure, which has been widely used in current SDP studies and makes a good balance between the precision and recall measures. AUC measure has also been widely used in current studies for SDP since it is insensitive to class imbalanced data and does not depend on an arbitrarily selected threshold …”

“…DEO is an optimizer, which is especially suitable for functions that may not be smooth or linear. It has been widely used in previous studies for hyperparameter optimization . According to a previous study, DEO has more competitive advantages when compared with other metaheuristic search algorithms, such as particle swarm optimization or genetic algorithm.…”

“…Table provides parameter names, their descriptions, and default values in the DEO phase. In this paper, we use the value of hyperparameters in differential evolution, which is suggested by Agrawal and Menzies . Table provides regularization parameter names, search ranges explored by DEO and their descriptions.…”

Applications of psychological science for actionable analytics

“…For building the defect predictors in this study, we elected to use Simple Logistic, Naive Bayes, Expectation Maximization, Support Vector Machine. We chose these learners because past studies shows that, for defect prediction tasks, these four learners represents four different levels of performance among a bunch of different learners [3,21]. Thus they are selected as the state-of-the-art learns to be compared with FFTs on the defect prediction data.…”