Effect of the Sampling of a Dataset in the Hyperparameter Optimization Phase over the Efficiency of a Machine Learning Algorithm

Abstract

Selecting the best configuration of hyperparameter values for a Machine Learning model yields directly in the performance of the model on the dataset. It is a laborious task that usually requires deep knowledge of the hyperparameter optimizations methods and the Machine Learning algorithms. Although there exist several automatic optimization techniques, these usually take significant resources, increasing the dynamic complexity in order to obtain a great accuracy. Since one of the most critical aspects in this computational consume is the available dataset, among others, in this paper we perform a study of the effect of using different partitions of a dataset in the hyperparameter optimization phase over the efficiency of a Machine Learning algorithm. Nonparametric inference has been used to measure the rate of different behaviors of the accuracy, time, and spatial complexity that are obtained among the partitions and the whole dataset. Also, a level of gain is assigned to each partition allowing us to study patterns and allocate whose samples are more profitable. Since Cybersecurity is a discipline in which the efficiency of Artificial Intelligence techniques is a key aspect in order to extract actionable knowledge, the statistical analyses have been carried out over five Cybersecurity datasets.

Vibration-Response-Only Structural Health Monitoring for Offshore Wind Turbine Jacket Foundations via Convolutional Neural Networks

“…To select the best architecture and to tune the different hyperparameters usually requires significant computational resources. As one of the most critical aspects of computational cost is the dataset size, in this paper, following the results presented in [27,28], the small dataset (without augmentation) is used to define the CNN architecture and quickly (coarse) tune the hyperparameters. Next, the obtained optimal hyperparameters for the small dataset are used as initial values to fine-tune the hyperparameters with the large dataset (with data augmentation).…”

Using machine learning to predict fire‐ignition occurrences from lightning forecasts

“…Thus, hyper‐parameter tuning is the problem of choosing a set of optimal hyper‐parameters for a learning algorithm (Ghawi and Pfeffer, 2019). To obtain good accuracy in a model, the easiest way is to test and compare different parameter combinations (DeCastro‐García et al ., 2019), and this is how hyper‐parameter tuning is approached. Hyper‐parameter tuning for the study was undertaken by creating different unique models with different sets of hyper‐parameters using tuning tools from the Python scikit‐learn package.…”

HyP-ABC: A Novel Automated Hyper-Parameter Tuning Algorithm Using Evolutionary Optimization

“…This leads to large search space scales, making them the most complex ML models to be tuned [19]. Furthermore, since different ML algorithms have different types of hyper-parameters (continuous, integer, and categorical), they should be treated differently in tuning processes [24]. Hence, they are the main focus of this study.…”

“…This is despite BO or PSO methods that may stick to a local optimum and which fail to reach a global optimum [6]. 3) Parallelization is one the advantages of PBA because each population can be assessed on one machine [19], while sequential methods such as BOs and GAs are challenging to parallelize since solutions are dependant on each other [24]. 4) Some PBA methods such as GA have some additional hyper-parameters (crossover and mutation probability, population size, number of generations, crossover, mutation, and selection operators) to tune; therefore, GA has lower convergence speed [39].…”