On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice

Abstract

Machine learning algorithms have been used widely in various applications and areas. To fit a machine learning model into different problems, its hyperparameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. It often requires deep knowledge of machine learning algorithms and appropriate hyper-parameter optimization techniques. Although several automatic optimization techniques exist, they have different strengths and drawbacks when applied to different types of problems. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization. This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyperparameter configurations effectively.

The Impact Of Twitter And News Count Variables On Stock Price Prediction Via Neural Networks

“…It is important to note that in the larger body of research that employs neural networks as the primary algorithm, there is a lack of standardization regarding the design. Furthermore, the vast majority of research focused on neural network design is more focused on hyperparameter optimization via algorithms rather than establishing empirical standards that researchers can utilize as proven baselines to effectively compare and further develop different neural networks (Yang & Shami, 2020). In this section, we will go over how each neural network design decision was made.…”

Review: Deep Learning in Electron Microscopy

“…Developers often optimize ANN hyperparameters by experimenting with a range of heuristic values. Hyperparameter optimization algorithms [1335][1336][1337][1338][1339][1340] can automate optimizer hyperparameter selection. However, automatic hyperparameter optimizers may not yield sufficient performance improvements relative to well-established heuristics to justify their use, especially in initial stages of development.…”

Predicting Model Training Time to Optimize Distributed Machine Learning Applications

“…The secondary goal is to determine which factors significantly influence this. Model hyperparameters [7] are investigated first, as the relationship between these and training time is often obvious. We aim not only to identify them but to model their influence on training time.…”