An Evolutionary Approach to Variational Autoencoders

Hajewski, Jeff; Oliveira, Suely

doi:10.1109/ccwc47524.2020.9031239

Cited by 9 publications

(15 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, over the past years little research has been undertaken to find the optimal depth and width in an unsupervised manner. To the best of our knowledge, no prior research has been carried out to address the automatic learning of AutoEncoders' depth in an unsupervised fashion except the evolutionary and genetic algorithms presented in [26,27]. The encoder and decoder layers of a stacked AutoEncoder can be fully connected or convolutional layers.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, the problem of learning the width and the depth of deep AEs has been addressed. As such, the topology of the AE is learned while training the model through dynamic expansion and\or pruning of neurons and/or the layers [18,[26][27][28][29][30][31]. These approaches are referred to as dynamic or evolving AE architecture in the literature.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Depth Learning in Stacked AutoEncoders

Alfayez,

Bchir,

Ben Ismail

2023

Applied Sciences

View full text Add to dashboard Cite

The effectiveness of deep learning models depends on their architecture and topology. Thus, it is essential to determine the optimal depth of the network. In this paper, we propose a novel approach to learn the optimal depth of a stacked AutoEncoder, called Dynamic Depth for Stacked AutoEncoders (DDSAE). DDSAE learns in an unsupervised manner the depth of a stacked AutoEncoder while training the network model. Specifically, we propose a novel objective function, aside from the AutoEncoder’s loss function to optimize the network depth: The optimization of the objective function determines the layers’ relevance weights. Additionally, we propose an algorithm that iteratively prunes the irrelevant layers based on the learned relevance weights. The performance of DDSAE was assessed using benchmark and real datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Dynamic Depth Learning in Stacked AutoEncoders

Alfayez,

Bchir,

Ben Ismail

2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…[107]- [110], [112], [114], [115], [118], [120], [121], [123], [125], [126], [128]- [133], [135], [137], [141], [149], [156], [158], [161], [162], [172], [173], [175], [183], [192], [194]- [197] 1…”

Section: B Comparison On Cifar-10 and Cifar-100unclassified

“…Such computational resources are not available for anyone who is interested in NAS. Retain the best group [26], [36], [38], [42], [49], [55], [73], [77], [84], [97], [111], [114], [132], [156], [157] Discard the worst or the oldest [56], [76], [86], [130], [131], [155], [184] Roulette [29], [30], [44], [60], [62], [107], [113], [116], [133] Tournament selection [9], [19], [20], [31], [59], [63], [80], [93], [99], [108], [117], [120], [125], [127], [128], [184] Others [57], [105] Almost all o...…”

Section: Shorten the Evaluation Timementioning

confidence: 99%

A Survey on Evolutionary Neural Architecture Search

Liu,

Sun,

Xue

et al. 2020

Preprint

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) have achieved great success in many applications, such as image classification, natural language processing and speech recognition. The architectures of DNNs have been proved to play a crucial role in its performance. However, designing architectures for different tasks is a difficult and time-consuming process of trial and error. Neural Architecture Search (NAS), which received great attention in recent years, can design the architecture automatically. Among different kinds of NAS methods, Evolutionary Computation (EC) based NAS methods have recently gained much attention and success. Unfortunately, there is not a comprehensive summary of the EC-based methods. This paper reviews 100+ papers of EC-based NAS methods in light of the common process. Four steps of the process have been covered in this paper including population initialization, population operators, evaluation and selection. Furthermore, current challenges and issues are also discussed to identify future research in this field.

show abstract

“…In today's literature, this problem is thus generally circumvented by basing the reward on a second model that judges the generated architecture through a supervised proxy objective, i.e. scoring the generated images according to a classifier (Gong et al, 2019), or by refraining from reinforcement learning and adopting techniques from evolutionary computing (Hajewski and Oliveira, 2020).…”

Section: Short-term Prospectsmentioning

confidence: 99%

Designing deep neural networks for continual learning in an open world

Mundt¹

View full text Add to dashboard Cite

Deep learning with neural networks seems to have largely replaced traditional design of computer vision systems. Automated methods to learn a plethora of parameters are now used in favor of previously practiced selection of explicit mathematical operators for a specific task. The entailed promise is that practitioners no longer need to take care of every individual step, but rather focus on gathering big amounts of data for neural network training. As a consequence, both a shift in mindset towards a focus on big datasets, as well as a wave of conceivable applications based exclusively on deep learning can be observed. This PhD dissertation aims to uncover some of the only implicitly mentioned or overlooked deep learning aspects, highlight unmentioned assumptions, and finally introduce methods to address respective immediate weaknesses. In the author’s humble opinion, these prevalent shortcomings can be tied to the fact that the involved steps in the machine learning workflow are frequently decoupled. Success is predominantly measured based on accuracy measures designed for evaluation with static benchmark test sets. Individual machine learning workflow components are assessed in isolation with respect to available data, choice of neural network architecture, and a particular learning algorithm, rather than viewing the machine learning system as a whole in context of a particular application. Correspondingly, in this dissertation, three key challenges have been identified: 1. Choice and flexibility of a neural network architecture. 2. Identification and rejection of unseen unknown data to avoid false predictions. 3. Continual learning without forgetting of already learned information. These latter challenges have already been crucial topics in older literature, alas, seem to require a renaissance in modern deep learning literature. Initially, it may appear that they pose independent research questions, however, the thesis posits that the aspects are intertwined and require a joint perspective in machine learning based systems. In summary, the essential question is thus how to pick a suitable neural network architecture for a specific task, how to recognize which data inputs belong to this context, which ones originate from potential other tasks, and ultimately how to continuously include such identified novel data in neural network training over time without overwriting existing knowledge. Thus, the central emphasis of this dissertation is to build on top of existing deep learning strengths, yet also acknowledge mentioned weaknesses, in an effort to establish a deeper understanding of interdependencies and synergies towards the development of unified solution mechanisms. For this purpose, the main portion of the thesis is in cumulative form. The respective publications can be grouped according to the three challenges outlined above. Correspondingly, chapter 1 is focused on choice and extendability of neural network architectures, analyzed in context of popular image classification tasks. An algorithm to automatically determine neural network layer width is introduced and is first contrasted with static architectures found in the literature. The importance of neural architecture design is then further showcased on a real-world application of defect detection in concrete bridges. Chapter 2 is comprised of the complementary ensuing questions of how to identify unknown concepts and subsequently incorporate them into continual learning. A joint central mechanism to distinguish unseen concepts from what is known in classification tasks, while enabling consecutive training without forgetting or revisiting older classes, is proposed. Once more, the role of the chosen neural network architecture is quantitatively reassessed. Finally, chapter 3 culminates in an overarching view, where developed parts are connected. Here, an extensive survey further serves the purpose to embed the gained insights in the broader literature landscape and emphasizes the importance of a common frame of thought. The ultimately presented approach thus reflects the overall thesis’ contribution to advance neural network based machine learning towards a unified solution that ties together choice of neural architecture with the ability to learn continually and the capability to automatically separate known from unknown data.

show abstract

An Evolutionary Approach to Variational Autoencoders

Cited by 9 publications

References 11 publications

Dynamic Depth Learning in Stacked AutoEncoders

Dynamic Depth Learning in Stacked AutoEncoders

A Survey on Evolutionary Neural Architecture Search

Designing deep neural networks for continual learning in an open world

Contact Info

Product

Resources

About