A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification

Wang, Bin; Sun, Yanan; Xue, Bing; Zhang, Mengjie

doi:10.1007/978-3-030-03991-2_24

Cited by 59 publications

(44 citation statements)

References 17 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Daha iyi değere sahip birey bir sonraki jenerasyona aktarılır [55]. [53] k: [1,8] c: [1,128] s: [1,4] k: [1,4] pa: max, avg s: [1,4] n: [1,2048] -DE Wang (2018) [56] k: [1,8] c: [1,128] s: [1,4] k: [1,4] pa: max, avg s: [1,4] n: […”

Section: Diferansiyel Gelişimunclassified

Konvolüsyonel Sinir Ağlarında Hiper-Parametre Optimizasyonu Yöntemlerinin İncelenmesi

Gülcü

Kuş

2019

Gazi Üniversitesi Fen Bilimleri Dergisi Part C: Tasarım Ve Teknoloji

View full text Add to dashboard Cite

In this study, we provide a review on the meta-heuristic methods like Genetic Algorithms, Particle Swarm Optimization, Differential Evolution and Bayes Optimization that have been used extensively to optimize hyper-parameters in Convolutional Neural Networks (CNN). We highlight the hyper-parameters that have been selected to be optimized in those studies along with the value domains of those parameters. These studies reveal that the number of layers, number of kernels and size of those kernels at each layer, learning rate and the batch size are among the hyper-parameters that affect the performance of the CNNs the most. Figure A. structure of convolutional neural networks Purpose: In this study, meta-heuristic methods that have been used to optimize convolutional neural networks are investigated. A performance comparison of these methods on different image datasets has been presented. The advantages and disadvantages of the optimization approaches have been presented with the aim of providing the user important points that should be considered during hyper-parameter selection process. Results: The definiton of "the best" set of hyper-parameters in convolutional neural networks depends on the problem or in this case, on the dataset. But it is clear from the studies that the selection of some parameters directly affect the performance of the networks. Number of layers, number of filters in each layer and size of each filter, regularization method, learning rate and batch size are among the most important parameters. It is easy to conclude that Genetic Algorithms (GA) are the most widely studied techniques used in hyper-parameter optimizaton. This is due to the fact that they yield successful results in most of the studies. While selecting the optimization method, one should consider the size of the problem, available computational budget and time. In addition, accuracy expectations should also be taken into account. For the problems with small hyper-parameter search space, methods like Grid Search would be sufficient, but for the problems with large search space, meta-heuristic methods would be more convenient. Conclusion: In this study, the effect of hyper-parameter optimization methods on classification performance is investigated. GA and Particle Swarm Optimization (PSO) methods are the two most-widely used meta-heuristics for hyper-parameter optimization. The computational burden of these methods can be justified with the accuracy improvement achieved with them. If the computational resources are limited, and it is desired to obtain good results in reasonable amount of time, then other methods like TPE and SMAC would be good choices.

show abstract

Section: Diferansiyel Gelişimunclassified

Konvolüsyonel Sinir Ağlarında Hiper-Parametre Optimizasyonu Yöntemlerinin İncelenmesi

Gülcü

Kuş

2019

Gazi Üniversitesi Fen Bilimleri Dergisi Part C: Tasarım Ve Teknoloji

View full text Add to dashboard Cite

show abstract

“…Assuno et al [32] have presented DENSER, a work where a multi-level encoding of candidate solutions allow for the optimization of the topology of the network and the activation functions, with authors claiming that it can be used also to evolve the hyperparameters of the learning process as well as of the data augmentation stage. In late 2018, Wang et al [33] used differential evolution to optimize different hyperparameters of both convolutional and fully-connected layers. Sun et al [34] have recently proposed the application of a genetic algorithm where two building blocks (ResNet and DenseNet) are used to evolve the CNN architecture.…”

Section: Complexitymentioning

confidence: 99%

Hybridizing Evolutionary Computation and Deep Neural Networks: An Approach to Handwriting Recognition Using Committees and Transfer Learning

2019

View full text Add to dashboard Cite

Neuroevolution is the field of study that uses evolutionary computation in order to optimize certain aspect of the design of neural networks, most often its topology and hyperparameters. The field was introduced in the late-1980s, but only in the latest years the field has become mature enough to enable the optimization of deep learning models, such as convolutional neural networks. In this paper, we rely on previous work to apply neuroevolution in order to optimize the topology of deep neural networks that can be used to solve the problem of handwritten character recognition. Moreover, we take advantage of the fact that evolutionary algorithms optimize a population of candidate solutions, by combining a set of the best evolved models resulting in a committee of convolutional neural networks. This process is enhanced by using specific mechanisms to preserve the diversity of the population. Additionally, in this paper, we address one of the disadvantages of neuroevolution: the process is very expensive in terms of computational time. To lessen this issue, we explore the performance of topology transfer learning: whether the best topology obtained using neuroevolution for a certain domain can be successfully applied to a different domain. By doing so, the expensive process of neuroevolution can be reused to tackle different problems, turning it into a more appealing approach for optimizing the design of neural networks topologies. After evaluating our proposal, results show that both the use of neuroevolved committees and the application of topology transfer learning are successful: committees of convolutional neural networks are able to improve classification results when compared to single models, and topologies learned for one problem can be reused for a different problem and data with a good performance. Additionally, both approaches can be combined by building committees of transferred topologies, and this combination attains results that combine the best of both approaches.

show abstract

“…CNN and GA have already been combined in recent studies [24], [25], but these studies have focused on using CNN for classifying images and used evolutionary algorithms to optimize its parameters or to generate the optimal CNN network. Other combinations, such as a combinatorial neural network trained by a stochastic search method [26], were presented to study the datasets from PJM and Spain.…”

Section: Introductionmentioning

confidence: 99%

“…On the other hand, evolutionary algorithms (EA) are members of the family of population-based algorithms, which were developed to find quasi-optimal solutions in any complex search field. Particle Swarm Optimizations (PSO) [4], [19], [22], [23] and Evolutionary Strategy [24], [35] are developed to solve optimization problems of continuous variables without any constraints. Although variants of PSO (such as Discrete PSO) can deal with binary or discrete variables, it is still difficult to deal with inequality constraints (e.g., maximum and minimum limits of a variable).…”

Section: Introductionmentioning

confidence: 99%

Locational Marginal Price Forecasting Using Deep Learning Network Optimized by Mapping-Based Genetic Algorithm

2020

View full text Add to dashboard Cite

The convolutional neural network (CNN) is commonly used in visual recognitions and classifications. However, CNN can also be applied as a forecaster that can extract features from spatiotemporal data. This paper proposes a 24h ahead electricity price forecasting method, which integrates CNN with an evolutionary algorithm and utilizes spatiotemporal data. The optimal structure of the CNN network for the locational marginal price (LMP) forecasting was obtained using a genetic algorithm (GA). A gene mapping scheme was initially encoded to represent the search space and the process of selection, mutation, and crossover eliminated structures that did not satisfy the validation fitness function and then competitive individuals were generated. The evolution process uses the root mean square error (RMSE) as the validation fitness function, which is optimzed by training the created CNN network. The proposed gene mapping scheme can be used to design an optimal CNN structure once the mapping between gene binary bits and parameters/hyperparameters of CNN is given. Day-ahead LMP and demand datasets from Pennsylvania-New Jersey-Maryland (PJM) power market were used to demonstrate the evolutionary capability of the proposed method and the finding of optimal CNN structures. Each studied dataset was grouped into 4 subsets corresponding to various seasonal characteristics (different types of situations in real life). Experimental results revealed that the proposed GA-CNN always yielded a higher forecasting accuracy and lower error rates than other forecasting methods. INDEX TERMS Convolutional neural network, deep learning, electricity price forecasting, genetic algorithm, locational marginal price.

show abstract

A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification

Cited by 59 publications

References 17 publications

Konvolüsyonel Sinir Ağlarında Hiper-Parametre Optimizasyonu Yöntemlerinin İncelenmesi

Konvolüsyonel Sinir Ağlarında Hiper-Parametre Optimizasyonu Yöntemlerinin İncelenmesi

Hybridizing Evolutionary Computation and Deep Neural Networks: An Approach to Handwriting Recognition Using Committees and Transfer Learning

Locational Marginal Price Forecasting Using Deep Learning Network Optimized by Mapping-Based Genetic Algorithm

Contact Info

Product

Resources

About