Neuro-Evolutionary Transfer Learning Through Structural Adaptation

ElSaid, AbdElRahman; Karnas, Joshua; Lyu, Zimeng; Krutz, Daniel E.; Ororbia, Alexander G.; Desell, Travis

doi:10.1007/978-3-030-43722-0_39

Cited by 10 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proposed work is utilizes components from the Evolutionary Exploration of Augmenting Memory Models (EXAMM) algorithm [31] as its core and utilizes its mutation, crossover and training operations in online scenarios. EXAMM is a distributed NE algorithm that evolves progressively larger RNNs for large-scale, multivariate, real-world TSF [14,15]. EXAMM evolves RNN architectures consisting of varying recurrent connections and memory cells through a series of mutation and crossover (reproduction) operations.…”

Section: Methodsmentioning

confidence: 99%

“…Generated offspring inherit their weights from their parents, which can significantly reduce the time needed for their training and evaluation [25]. It has been shown that EXAMM can swiftly adapt RNNs in transfer learning scenarios, even when the input and output data streams are changed [15] [14], which served as a preliminary motivation and justification for being able to adapt and evolve RNNs for TSF in online scenarios.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

ONE-NAS: An Online NeuroEvolution based Neural Architecture Search for Time Series Forecasting

Lyu¹,

Desell²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Time series forecasting (TSF) is one of the most important tasks in data science, as accurate time series (TS) predictions can drive and advance a wide variety of domains including finance, transportation, health care, and power systems. However, real-world utilization of machine learning (ML) models for TSF suffers due to pretrained models being able to learn and adapt to unpredictable patterns as previously unseen data arrives over longer time scales. To address this, models must be periodically retained or redesigned, which takes significant human and computational resources. This work presents the Online NeuroEvolution based Neural Architecture Search (ONE-NAS) algorithm, which to the authors' knowledge is the first neural architecture search algorithm capable of automatically designing and training new recurrent neural networks (RNNs) in an online setting. Without any pretraining, ONE-NAS utilizes populations of RNNs which are continuously updated with new network structures and weights in response to new multivariate input data. ONE-NAS is tested on real-world large-scale multivariate wind turbine data as well a univariate Dow Jones Industrial Average (DJIA) dataset, and is shown to outperform traditional statistical time series forecasting, including naive, moving average, and exponential smoothing methods, as well as state of the art online ARIMA strategies. CCS CONCEPTS• Theory of computation → Online algorithms; • Computing methodologies → Neural networks; • Applied computing → Forecasting.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

ONE-NAS: An Online NeuroEvolution based Neural Architecture Search for Time Series Forecasting

Lyu¹,

Desell²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We chose eRNN for our TVA-E process due to its demonstrated ability to make accurate and reliable predictions using limited amounts of data in comparison with alternative options [14][15][16][17]. However, a myriad of alternative prediction techniques could be easily incorporated into our process to account for tactic volatility.…”

Section: Proposed Tactic Volatility Aware Processmentioning

confidence: 99%

Addressing Tactic Volatility in Self-Adaptive Systems Using Evolved Recurrent Neural Networks and Uncertainty Reduction Tactics

Ul¹,

Deshpande²,

ElSaid³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Self-adaptive systems frequently use tactics to perform adaptations. Tactic examples include the implementation of additional security measures when an intrusion is detected, or activating a cooling mechanism when temperature thresholds are surpassed. Tactic volatility occurs in real-world systems and is defined as variable behavior in the attributes of a tactic, such as its latency or cost. A system's inability to effectively account for tactic volatility adversely impacts its efficiency and resiliency against the dynamics of real-world environments. To enable systems' efficiency against tactic volatility, we propose a Tactic Volatility Aware (TVA-E) process utilizing evolved Recurrent Neural Networks (eRNN) to provide accurate tactic predictions. TVA-E is also the first known process to take advantage of uncertainty reduction tactics to provide additional information to the decision-making process and reduce uncertainty. TVA-E easily integrates into popular adaptation processes enabling it to immediately benefit a large number of existing self-adaptive systems. Simulations using 52,106 tactic records demonstrate that: I) eRNN is an effective prediction mechanism, II) TVA-E represents an improvement over existing state-of-the-art processes in accounting for tactic volatility, and III) Uncertainty reduction tactics are beneficial in accounting for tactic volatility. The developed dataset and tool can be found at https://tacticvolatility.github.io/ CCS CONCEPTS• Theory of computation → Semi-supervised learning; • Computing methodologies → Planning under uncertainty; Semisupervised learning settings; Genetic algorithms.

show abstract

“…The crucial aspect of the SLM approach is the geometric semantic mutation. ElSaid et al [36] proposed an approach based on [35], called network-aware adaptive structure transfer learning strategy with the goal of improving training time for deep RNNs. The authors used statistical information about the topology of the "source RNN" topology and the weight distributions.…”

Section: E DL Architecture: Deep Belief Networkmentioning

confidence: 99%

Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

Galván

Mooney

2021

IEEE Trans. Artif. Intell.

View full text Add to dashboard Cite

A variety of methods have been applied to the architectural configuration and learning or training of artificial deep neural networks (DNN). These methods play a crucial role in the success or failure of the DNN for most problems and applications. Evolutionary algorithms (EAs) are gaining momentum as a computationally feasible method for the automated optimization of DNNs. Neuroevolution is a term, which describes these processes of automated configuration and training of DNNs using EAs. While many works exist in the literature, no comprehensive surveys currently exist focusing exclusively on the strengths and limitations of using neuroevolution approaches in DNNs. Absence of such surveys can lead to a disjointed and fragmented field preventing DNNs researchers potentially adopting neuroevolutionary methods in their own research, resulting in lost opportunities for wider application within real-world deep learning problems. This article presents a comprehensive survey, discussion, and evaluation of the state-of-the-art in using EAs for architectural configuration and training of DNNs. This article highlights the most pertinent current issues and challenges in neuroevolution and identifies multiple promising future research directions. Impact Statement-The concept of deep learning originated from the study of artificial neural networks (ANNs). ANNs have achieved extraordinary results in a variety of diverse application areas. Numerous methods have been applied to the architectural configuration and learning or training of artificial DNN and these methods play a crucial role in the success or failure of the DNN for most problems and applications. Recently, EAs have been gaining momentum as a computationally feasible method (called neuroevolution) for the automated configuration and learning or training of DNNs. This article reviews over 170 recent scientific papers describing how major EAs paradigms are being applied by researchers to the configuration and optimization of multiple DNNs. By articulating a clear understanding of the context, state-of-the-art, and feasibility of Neuroevolution, researchers in AI, EAs, and DNN will benefit from this article. The impact of this article comes from contributing toward enhancing research capacity, knowledge, and skills for researchers currently working in neuroevolution and actively engaging those considering becoming involved in this area.

show abstract

Neuro-Evolutionary Transfer Learning Through Structural Adaptation

Cited by 10 publications

References 11 publications

ONE-NAS: An Online NeuroEvolution based Neural Architecture Search for Time Series Forecasting

ONE-NAS: An Online NeuroEvolution based Neural Architecture Search for Time Series Forecasting

Addressing Tactic Volatility in Self-Adaptive Systems Using Evolved Recurrent Neural Networks and Uncertainty Reduction Tactics

Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

Contact Info

Product

Resources

About