On the Performance of Differential Evolution for Hyperparameter Tuning

Schmidt, Mischa; Safarani, Shahd; Gastinger, Julia; Jacobs, Tobias; Nicolas, Sébastien; Schülke, Anett

doi:10.1109/ijcnn.2019.8851978

Cited by 20 publications

(11 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, the EA has achieved state-of-the-art performances in optimizing deep neural networks [10]- [13], [15]- [20]. It has also proved its superiority over Bayesian optimization [20], [40], [41], which is not applicable to solving a sequential decision problem such as the architecture of neural networks [38]. Moreover, the performance of Bayesian optimization is poor when using a high number of hyperparameters [13].…”

Section: A Optimization Of Hyperparameters and Architecturementioning

confidence: 99%

An Automatic Convolutional Neural Network Optimization Using a Diversity-Guided Genetic Algorithm

Fatyanosa

Aritsugi

2021

IEEE Access

View full text Add to dashboard Cite

Hyperparameters and architecture greatly influence the performance of convolutional neural networks (CNNs); therefore, their optimization is important to obtain the desired results. One of the state-ofthe-art methods to achieve this is the use of neuroevolution that utilizes a genetic algorithm (GA) to optimize a CNN. However, the GA is often trapped into a local optimum resulting in premature convergence. In this study, we propose an approach called the "diversity-guided genetic algorithm-convolutional neural network (DGGA-CNN)" that uses adaptive parameter control and random injection to facilitate the search process by exploration and exploitation while preserving the population diversity. The alternation between exploration and exploitation is guided by using an average pairwise Hamming distance. Moreover, the DGGA fully handles the architecture of the CNN by using a novel finite state machine (FSM) combined with three novel mutation mechanisms that are specifically created for architecture chromosomes. Tests conducted on suggestion mining and twitter airline datasets reveal that the DGGA-CNN performs well with valid architectures and a comparison with other methods demonstrates its capability and efficiency.

show abstract

Section: A Optimization Of Hyperparameters and Architecturementioning

confidence: 99%

An Automatic Convolutional Neural Network Optimization Using a Diversity-Guided Genetic Algorithm

Fatyanosa

Aritsugi

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…In addition to the internal parameters adjusted in the training step, ML methods are generally sensitive to hyperparameter definitions [26]. We propose a hybrid approach coupling the ML methods with hyperparameters tuning through a Randomized Search (RS) strategy in this work.…”

Section: Randomized Search (Rs) Strategymentioning

confidence: 99%

Gaussian processes regression for cyclodextrin host-guest binding prediction

Carvalho

Rosa

Gomes

et al. 2021

J Incl Phenom Macrocycl Chem

View full text Add to dashboard Cite

Machine Learning (ML) techniques are becoming an integral part of rational drug design and discovery. Data-driven modeling regularly outperforms physics-based models for predicting molecular binding affinities, placing ML as a promising tool. Cyclodextrins are nano-cages used to improve the delivery of insoluble or toxic drugs. Due to chemical similarity to proteins, ML approaches could vastly profit to improve affinity prediction and enhance their carriable drug portfolio. Here we evaluate the performance of three well-known ML methods -Support Vector Regression (SVR), Gaussian Process Regression (GPR), and eXtreme Gradient Boosting (XGB) -to predict the binding affinity of cyclodextrin and known ligands. We perform hyperparameter tuning through Random Search. The results were compatible with the presented literature. We were able to increase our previous prediction performance and present a GPR model to adjust to the data (R 2 = 0.803) with low prediction errors (RMSE = 1.811 kJ/mol and MAE = 1.201 kJ/mol).

show abstract

“…This work leverages the traces of experiments in [3], which executed hyperparameter tuning for six base learners by an evolutionary strategy. Running different algorithm selection policies on the recorded experiment traces allows evaluating different bandit policies based on a common ground truth.…”

Section: B Experimental Validationmentioning

confidence: 99%

“…That enables a fair comparison among the bandits: more expensive bandit strategies have to offset their higher computational costs by selecting better actions. 1) Computational Resources and Setup: [3] executed each tuner (and base learner) in a single docker container with only a single CPU core accessible. Parallel execution of different experiments was limited to ensure that a full CPU core was available for each docker container.…”

Section: B Experimental Validationmentioning

confidence: 99%

“…The combination of learning curve extrapolation and accounting for computation time improves the performance of multi-armed bandits in algorithm selection problems. Our empirical evaluation uses the 99 traces for six different base learners from [3]. The evaluation shows that even a simple approach to fit learning curves provides gains for tight time budgets.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

HAMLET -- A Learning Curve-Enabled Multi-Armed Bandit for Algorithm Selection

Schmidt¹,

Gastinger²,

Nicolas³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Automated algorithm selection and hyperparameter tuning facilitates the application of machine learning. Traditional multi-armed bandit strategies look to the history of observed rewards to identify the most promising arms for optimizing expected total reward in the long run. When considering limited time budgets and computational resources, this backward view of rewards is inappropriate as the bandit should look into the future for anticipating the highest final reward at the end of a specified time budget. This work addresses that insight by introducing HAMLET, which extends the bandit approach with learning curve extrapolation and computation time-awareness for selecting among a set of machine learning algorithms. Results show that the HAMLET Variants 1-3 exhibit equal or better performance than other bandit-based algorithm selection strategies in experiments with recorded hyperparameter tuning traces for the majority of considered time budgets. The best performing HAMLET Variant 3 combines learning curve extrapolation with the well-known upper confidence bound exploration bonus. That variant performs better than all non-HAMLET policies with statistical significance at the 95% level for 1,485 runs.

show abstract

On the Performance of Differential Evolution for Hyperparameter Tuning

Cited by 20 publications

References 9 publications

An Automatic Convolutional Neural Network Optimization Using a Diversity-Guided Genetic Algorithm

An Automatic Convolutional Neural Network Optimization Using a Diversity-Guided Genetic Algorithm

Gaussian processes regression for cyclodextrin host-guest binding prediction

HAMLET -- A Learning Curve-Enabled Multi-Armed Bandit for Algorithm Selection

Contact Info

Product

Resources

About