Predicting Model Training Time to Optimize Distributed Machine Learning Applications

Abstract

Despite major advances in recent years, the field of Machine Learning continues to face research and technical challenges. Mostly, these stem from big data and streaming data, which require models to be frequently updated or re-trained, at the expense of significant computational resources. One solution is the use of distributed learning algorithms, which can learn in a distributed manner, from distributed datasets. In this paper, we describe CEDEs—a distributed learning system in which models are heterogeneous distributed Ensembles, i.e., complex models constituted by different base models, trained with different and distributed subsets of data. Specifically, we address the issue of predicting the training time of a given model, given its characteristics and the characteristics of the data. Given that the creation of an Ensemble may imply the training of hundreds of base models, information about the predicted duration of each of these individual tasks is paramount for an efficient management of the cluster’s computational resources and for minimizing makespan, i.e., the time it takes to train the whole Ensemble. Results show that the proposed approach is able to predict the training time of Decision Trees with an average error of 0.103 s, and the training time of Neural Networks with an average error of 21.263 s. We also show how results depend significantly on the hyperparameters of the model and on the characteristics of the input data.

Biomass Higher Heating Value Estimation: A Comparative Analysis of Machine Learning Models

“…The SVM and RFR models also have a shorter execution time due to the lower complexity of the model and the complexity of the optimization process. The experiments conducted by Guimarães et al (2023) [45] demonstrate that predicting the training time of models like decision trees and neural networks is achievable with reasonable accuracy. The average prediction error is 0.103 s for decision trees and 21.263 s for ANNs, which is acceptable considering their training times span up to 14 s and over 1400 s, respectively.…”