A novel online supervised hyperparameter tuning procedure applied to cross-company software effort estimation

Abstract

Software effort estimation is an online supervised learning problem, where new training projects may become available over time. In this scenario, the Cross-Company (CC) approach Dycom can drastically reduce the number of Within-Company (WC) projects needed for training, saving their collection cost. However, Dycom requires CC projects to be split into subsets. Both the number and composition of such subsets can affect Dycom's predictive performance. Even though clustering methods could be used to automatically create CC subsets, there are no procedures for automatically tuning the number of clusters over time in online supervised scenarios. This paper proposes the first procedure for that. An investigation of Dycom using six clustering methods and three automated tuning procedures is performed, to check whether clustering with automated tuning can create well performing CC splits. A case study with the ISBSG Repository shows that the proposed tuning procedure in combination with a simple threshold-based clustering method is the most successful in enabling Dycom to drastically reduce (by a factor of 10) the number of required WC training projects, while maintaining (or even improving) predictive performance in comparison with a corresponding WC model. A detailed analysis is provided to understand the conditions under which this approach does or does not work well. Overall, the proposed online supervised tuning procedure was generally successful in enabling a very simple thresholdbased clustering approach to obtain the most competitive Dycom results. This demonstrates the value of automatically tuning hyperparameters over time in a supervised way.

Using machine learning to generate test oracles: a systematic literature review

“…Many ML techniques have a number of hyperparameters that can be tuned (e.g., the learning rate, number of hidden units, or activation function) [23]. Hyperparameter tuning can have a major impact on model accuracy, and can enable significant improvements in the results of even simple ML techniques.…”

Hyperparameter self-tuning for data streams

“…In terms of online hyperparameter tuning algorithms, there are few works [17,[23][24][25][26] that use support vector machines together with batch processing, gradient solutions combined with brute force or genetic algorithms to optimise hyperparameters. Lawal and Abdulkarim [23] introduce an incremental learning-model selection method for data stream batches.…”

“…The algorithm computes the hyperparameter gradients on the fly whenever a new datum is observed and, then, updates smoothly the hyperparameters with the average of the past and current hypergradients. Minku [25] proposed an online hyperparameter tuning method that maintains a number of model instances created from different subsets. The method applies computational brute force to find the model instance with the smallest validation error.…”

A heterogeneous online learning ensemble for non-stationary environments

“…In terms of how to set the parameters in real world problems, the difficulty is that the best values may change over time. Potentially, one could run multiple versions of the approach with different parameter settings [61]. The parameters 'β', 'θ' and 'Period' were analysed and their effect on prediction accuracy, ensemble size and drift detections.…”