Convolutional Neural Network Hyper-Parameters Optimization based on Genetic Algorithms

Abstract

In machine learning for computer vision based applications, Convolutional Neural Network (CNN) is the most widely used technique for image classification. Despite these deep neural networks efficiency, choosing their optimal architecture for a given task remains an open problem. In fact, CNNs performance depends on many hyper-parameters namely CNN depth, convolutional layer number, filters number and their respective sizes. Many CNN structures have been manually designed by researchers and then evaluated to verify their efficiency. In this paper, our contribution is to propose an innovative approach, labeled Enhanced Elite CNN Model Propagation (Enhanced E-CNN-MP), to automatically learn the optimal structure of a CNN. To traverse the large search space of candidate solutions our approach is based on Genetic Algorithms (GA). These meta-heuristic algorithms are wellknown for non-deterministic problem resolution. Simulations demonstrate the ability of the designed approach to compute optimal CNN hyper-parameters in a given classification task. Classification accuracy of the designed CNN based on Enhanced E-CNN-MP method, exceed that of public CNN even with the use of the Transfer Learning technique. Our contribution advances the current state by offering to scientists, regardless of their field of research, the ability of designing optimal CNNs for any particular classification problem.

Automating Configuration of Convolutional Neural Network Hyperparameters Using Genetic Algorithm

“…We used our algorithm to find optimal structures for the datasets MNIST, CIFAR10, and CALTECH256, running each test 10 times with different seeds for the genetic algorithm. Besides our proposal, we also performed tests using the algorithms proposed by Loussaief et al [29], Sun et al [39], and Bhandare et al [3] for each dataset. For Loussaief et al we were able to execute 10 different runs, while for the rest only one run was performed because of time constraints.…”

“…Sun et al's algorithms yields 99.3% accuracy after 135.42 total hours, while Bhandare's algorithm achieves 99.02% accuracy after 6.4 hours. We can observe the evolution of the validation accuracies of the 10 elites of our proposal through the generations in Fig 6. MNIST CIFAR10 Caltech256 Loussaief [29] 99 The second dataset we worked with is the CIFAR10 dataset. It is composed of 60,000 32x32 RGB images labeled in 10 categories, such as dog, truck, or ship.…”

“…In [29], the authors work with the more demanding Caltech-256 dataset. For that task, they optimize the kernel size and kernel number maintaining a fixed depth reporting positive results.…”

“…Other researches have used more simple CNN architectures [3], [30] and genetic algorithms to optimize not only the structure of the network, but the learning parameters as well, which entails comparisons between trained and untrained networks during the process. Most current developments [3], [29], [30] use traditional crossover based on genetic algorithms operators, destroying the structure of the parent networks as offspring population is created, ignoring the nature of the underlying Machine Learning (ML) problem.…”

Optimizing the neural network hyperparameters utilizing genetic algorithm

“…The authors improved their results by combining CNN with GA to find the optimal CNN parameters, leading to more accurate image classification. In another image recognition study carried out by Loussaief and Abdelkrim (2018), the numbers of convolutional layers and filters, as well as their sizes in each layer, were optimized using GA. They managed to increase the classification accuracy of the CNN from 90% to >98%.…”