Evolving Deep Neural Networks

Miikkulainen, Risto; Liang, Jason; Meyerson, Elliot; Rawal, Aditya; Fink, Dan; Francon, Olivier; Raju, Bala; Shahrzad, Hormoz; Navruzyan, Arshak; Duffy, Nigel; Hodjat, Babak

doi:10.48550/arxiv.1703.00548

Cited by 54 publications

(85 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this case, the genotype represents an abstraction for the implementation of a neural network. This representation can be direct, i.e., all nodes and connections of the neural architecture are encoded [22,28], or indirect, i.e., rules are specified to derive the concrete implementation of neural networks, such as in structured grammatical evolution [3,19].…”

Section: Neuroevolutionmentioning

confidence: 99%

“…The genotype is a direct representation of the neural network, where NEAT defines two lists for the genome of individuals: a list of neurons and a list of connections between these neurons. A further expansion of NEAT was proposed to enable larger search spaces in DeepNEAT and CoDeepNEAT [22]. In these models, the genes composing a genome are abstractions of entire layers, enabling the representation of deep neural networks.…”

Section: Neuroevolutionmentioning

confidence: 99%

“…Deep neural networks became popular and achieved strong performance in several tasks. Thus, the need for automation became relevant to improve progress on deeper models [22].…”

Section: Neuroevolutionmentioning

confidence: 99%

“…Neuroevolution is an approach used to design and optimize neural networks through the application of evolutionary algorithms [22,28,33]. These algorithms are based on the evolutionary mechanism found in nature, evolving a population of individuals through selective pressure, leading to the discovery of efficient solutions for a certain problem [27].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Demonstrating the Evolution of GANs Through t-SNE

Costa¹,

Lourenço²,

Correia³

et al. 2021

Applications of Evolutionary Computation

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) are powerful generative models that achieved strong results, mainly in the image domain. However, the training of GANs is not trivial, presenting some challenges tackled by different strategies. Evolutionary algorithms, such as COEGAN, were recently proposed as a solution to improve the GAN training, overcoming common problems that affect the model, such as vanishing gradient and mode collapse. In this work, we propose an evaluation method based on t-distributed Stochastic Neighbour Embedding (t-SNE) to assess the progress of GANs and visualize the distribution learned by generators in training. We propose the use of the feature space extracted from trained discriminators to evaluate samples produced by generators and from the input dataset. A metric based on the resulting t-SNE maps and the Jaccard index is proposed to represent the model quality. Experiments were conducted to assess the progress of GANs when trained using COEGAN. The results show both by visual inspection and metrics that the Evolutionary Algorithm gradually improves discriminators and generators through generations, avoiding problems such as mode collapse.

show abstract

Section: Neuroevolutionmentioning

confidence: 99%

Section: Neuroevolutionmentioning

confidence: 99%

“…Deep neural networks became popular and achieved strong performance in several tasks. Thus, the need for automation became relevant to improve progress on deeper models [22].…”

Section: Neuroevolutionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Demonstrating the Evolution of GANs Through t-SNE

Costa¹,

Lourenço²,

Correia³

et al. 2021

Applications of Evolutionary Computation

View full text Add to dashboard Cite

show abstract

“…Importantly, recognising the advantages of evolution as a global optimiser, there has been a paradigm shift towards utilising NE as an optimiser for the network structure in combination with backpropagation (BP) to fine-tune the network weights. For instance, deep convolutional NNs (CNNs) with multiple layers and millions of parameters have been evolved for tasks ranging from image classification [16], [17], image captioning [17] (using an evolved deep Long Short-Term Memory (LSTM) network) and even applications in particle physics (neutron scattering model selection) [18]. A differentiable version of CPPN was proposed in [19] to efficiently compress the representation of deep CNNs.…”

Section: Imentioning

confidence: 99%

Epigenetic evolution of deep convolutional models

Hadjiivanov

Blair

2019

2019 IEEE Congress on Evolutionary Computation (CEC)

View full text Add to dashboard Cite

In this study, we build upon a previously proposed neuroevolution framework to evolve deep convolutional models. Specifically, the genome encoding and the crossover operator are extended to make them applicable to layered networks. We also propose a convolutional layer layout which allows kernels of different shapes and sizes to coexist within the same layer, and present an argument as to why this may be beneficial. The proposed layout enables the size and shape of individual kernels within a convolutional layer to be evolved with a corresponding new mutation operator. The proposed framework employs a hybrid optimisation strategy involving structural changes through epigenetic evolution and weight update through backpropagation in a population-based setting. Experiments on several image classification benchmarks demonstrate that the crossover operator is sufficiently robust to produce increasingly performant offspring even when the parents are trained on only a small random subset of the training dataset in each epoch, thus providing direct confirmation that learned features and behaviour can be successfully transferred from parent networks to offspring in the next generation.

show abstract

AMC: AutoML for Model Compression and Acceleration on Mobile Devices

Lin

Liu

et al. 2018

Computer Vision – ECCV 2018

989

842

View full text Add to dashboard Cite

Model compression is a critical technique to efficiently deploy neural network models on mobile devices which have limited computation resources and tight power budgets. Conventional model compression techniques rely on hand-crafted heuristics and rule-based policies that require domain experts to explore the large design space trading off among model size, speed, and accuracy, which is usually sub-optimal and time-consuming. In this paper, we propose AutoML for Model Compression (AMC) which leverage reinforcement learning to provide the model compression policy. This learning-based compression policy outperforms conventional rule-based compression policy by having higher compression ratio, better preserving the accuracy and freeing human labor. Under 4× FLOPs reduction, we achieved 2.7% better accuracy than the handcrafted model compression policy for VGG-16 on ImageNet. We applied this automated, push-the-button compression pipeline to MobileNet and achieved 1.81× speedup of measured inference latency on an Android phone and 1.43× speedup on the Titan XP GPU, with only 0.1% loss of ImageNet Top-1 accuracy. Reward= -Error*log(FLOP) Agent: DDPG Action: Compress with Sparsity ratio at (e.g. 50%) Embedding st=[N,C,H,W,i…] Environment: Channel Pruning Layer t-1 Layer t Layer t+1 Critic Actor Embedding Original NN Model Compression by Human: Labor Consuming, Sub-optimal Model Compression by AI: Automated, Higher Compression Rate, Faster Compressed NN AMC Engine Original NN Compressed NN 30% 50% ? %

show abstract

Evolving Deep Neural Networks

Cited by 54 publications

References 0 publications

Demonstrating the Evolution of GANs Through t-SNE

Demonstrating the Evolution of GANs Through t-SNE

Epigenetic evolution of deep convolutional models

AMC: AutoML for Model Compression and Acceleration on Mobile Devices

Contact Info

Product

Resources

About