An empirical comparison of validation methods for software prediction models

Abstract

Model validation methods (e.g., k-fold cross-validation) use historical data to predict how well an estimation technique (e.g., random forest) performs on the current (or future) data. Studies in the contexts of software development effort estimation (SDEE) and software fault prediction (SFP) have used and investigated different model validation methods. However, no conclusive indications to suggest which model validation method has a major impact on the prediction accuracy and stability of estimation techniques. Some studies have investigated model validation methods specific to data about either SDEE or SFP. To the best of our knowledge, there is no study in the literature, which has employed different validation methods both with SDEE and SFP data. The aim of this paper is to consider different methods (10) from the family of cross-validation (CV) and bootstrap validation methods to identify which one contributes to obtaining a better prediction accuracy for both types of data. We also evaluate which model validation methods allow the estimation techniques to provide stable performances (i.e., with lower variance). To this aim, we present an empirical study involving six datasets from the domain of SDEE and six datasets from the SFP domain. The results reveal that repeated 10-fold CV with SDEE and optimistic boot with SFP data are the model validation methods that provide a better prediction accuracy in a greater number of experiments than the other model validation methods. Furthermore, a model validation method can improve the prediction accuracy up to 60% with SDEE data and up to 36% when employing SFP data. The analysis also reveals that repeated fivefold CV produces more stable performances when the experiments are repeated on the same data.

Software Fault Prediction Using an RNN-Based Deep Learning Approach and Ensemble Machine Learning Techniques

“…Examples of hyper parameters include the number of epochs, batch size, and learning rate. Adjusting these hyper parameters may significantly impact the performance of a machine learning model and finding optimal values for a given problem is typically a critical step in the model development procedure [45]. Several tests were conducted before the current study to determine the appropriate value range.…”

Software fault prediction using deep learning techniques

“…The selection of appropriate parameters is critical and a complicated aspect of network training due to constraints such as memory limitations, trade-offs are inherently present in parameter selection [3], [47]. Throughout the examination, several hyper-parameters to measure the influence of accuracy.…”

“…Nested K fold cross validation is another way to tune parameters of an algorithms. Data is divided into k fold and one fold is reserved for test [3]. K-1 training folds used for validation.…”

Optimizing complexity weight parameter of use case points estimation using particle swarm optimization

“…Model validation is the process where the trained model is evaluated with a testing data set to foresee how good the performance of the estimation method [59]. The testing data set is a separate portion of the same data set from which the training set is derived.…”