An improved hyperparameter optimization framework for AutoML systems using evolutionary algorithms

Abstract

Machine learning strategy has changed the face of automated models by integrating themselves into many application domains. The spectrum of applications ranges over various domains, from atmospheric analysis to medical diagnosis. All these applications are design-sensitive, implying that the model's performance depends highly on the selected machine learning algorithm, training procedures, regularization methods, and most importantly, how the hyperparameters are tuned. With the advent of AutoML systems, all these processes including hyperparameter optimization can be automated, producing better performance and faster results. The ideal hyperparameter setting for machine learning models has a direct and significant impact on the model's performance. This paper studies different AutoML models and the hyperparameter techniques used by them for image classification problems. We also discuss some of the libraries for hyperparameter optimization and analyse how they work for different image classification problems. Moreover, the paper proposes a framework for hyperparameter optimization that combines Bayesian optimization and evolutionary algorithms. Experiments are carried out on image classification benchmark datasets to assess the performance of different optimization approaches including the proposed model.

Hyperparameter Optimization for Deep NeuralNetwork Models: A Comprehensive Study onMethods and Techniques

“…The paper [5] proposed a hybrid optimization method called STEADE, which initially started to estimate the objective function by using the concept of Radial Basis Function interpolation, and then transfer the knowledge to an Evolutionary Algorithm (EA) method, which is used generate new solution domain by utilizing the Bayesian Optimization framework. The paper [34] proposed a hyperparameter optimization method that uses the concept of covariance matrix adaptation-evolutionary strategy and differential evolution to improve the performance of standard Bayesian optimization. We have summarized the various HPO methods in Table 1…”