Improving neuroevolutionary transfer learning of deep recurrent neural networks through network-aware adaptation

ElSaid, AbdElRahman; Karns, Joshua; Lyu, Zimeng; Krutz, Daniel E.; Ororbia, Alexander G.; Desell, Travis

doi:10.1145/3377930.3390193

Cited by 14 publications

(7 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…while the other weights are copied from the parent. This network-aware approach using the statistical distribution of a network's weights has also been shown in other work to speed transfer learning, lending further credence to this approach (ElSaid et al 2020).…”

Section: Weight Initialization and Inheritancesupporting

confidence: 60%

An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution

Lyu,

ElSaid,

Karns

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Weight initialization is critical in being able to successfully train artificial neural networks (ANNs), and even more so for recurrent neural networks (RNNs) which can easily suffer from vanishing and exploding gradients. In neuroevolution, where evolutionary algorithms are applied to neural architecture search, weights typically need to be initialized at three different times: when initial genomes (ANN architectures) are created at the beginning of the search, when offspring genomes are generated by crossover, and when new nodes or edges are created during mutation. This work explores the difference between using Xavier, Kaiming, and uniform random weight initialization methods, as well as novel Lamarckian weight inheritance methods for initializing new weights during crossover and mutation operations. These are examined using the Evolutionary eXploration of Augmenting Memory Models (EXAMM) neuroevolution algorithm, which is capable of evolving RNNs with a variety of modern memory cells (e.g., LSTM, GRU, MGU, UGRNN and Delta-RNN cells) as well recurrent connections with varying time skips through a high performance island based distributed evolutionary algorithm. Results show that with statistical significance, utilizing the Lamarckian strategies outperforms Kaiming, Xavier and uniform random weight initialization, and can speed neuroevolution by requiring less backpropagation epochs to be evaluated for each generated RNN.

show abstract

Section: Weight Initialization and Inheritancesupporting

confidence: 60%

An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution

Lyu,

ElSaid,

Karns

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The main objects of tourism security early warning system are tourists or local residents [ 15 , 16 ]. Therefore, the information it provides is more detailed, which has a good effect in the security of outbound tourism.…”

Section: Related Workmentioning

confidence: 99%

[Retracted] Optimization Algorithm of Tourism Security Early Warning Information System Based on Long Short‐Term Memory (LSTM)

Feng

2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Tourism safety is the focus of the tourism industry. It is not only related to the safety of tourists’ lives and property, but also related to social stability and sustainable development of the tourism industry. However, the security early warning of many scenic spots focuses on the response measures and remedial plans after the occurrence of security incidents, and the staff of many scenic spots have limited security awareness and information analysis ability, which is prone to lag in information release, and do not pay attention to the information of potential security problems. Therefore, this paper studies the optimization algorithm of the tourism security early warning information system based on the LSTM model and uses the recurrent neural network and LSTM to improve the processing and prediction ability of time-series data. The experimental results show that the number of three hidden layers in the tourism security early warning information system based on the LSTM model can reduce the training time of the model and improve the performance. Compared with the tourism safety early warning information system based on the BP neural network, it has better accuracy and stability, has better processing and prediction ability for time series data, and can monitor and analyze data scientifically in real-time and dynamically analyze data.

show abstract

“…The proposed work is utilizes components from the Evolutionary Exploration of Augmenting Memory Models (EXAMM) algorithm [31] as its core and utilizes its mutation, crossover and training operations in online scenarios. EXAMM is a distributed NE algorithm that evolves progressively larger RNNs for large-scale, multivariate, real-world TSF [14,15]. EXAMM evolves RNN architectures consisting of varying recurrent connections and memory cells through a series of mutation and crossover (reproduction) operations.…”

Section: Methodsmentioning

confidence: 99%

“…Generated offspring inherit their weights from their parents, which can significantly reduce the time needed for their training and evaluation [25]. It has been shown that EXAMM can swiftly adapt RNNs in transfer learning scenarios, even when the input and output data streams are changed [15] [14], which served as a preliminary motivation and justification for being able to adapt and evolve RNNs for TSF in online scenarios.…”

Section: Methodsmentioning

confidence: 99%

ONE-NAS: An Online NeuroEvolution based Neural Architecture Search for Time Series Forecasting

Lyu¹,

Desell²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Time series forecasting (TSF) is one of the most important tasks in data science, as accurate time series (TS) predictions can drive and advance a wide variety of domains including finance, transportation, health care, and power systems. However, real-world utilization of machine learning (ML) models for TSF suffers due to pretrained models being able to learn and adapt to unpredictable patterns as previously unseen data arrives over longer time scales. To address this, models must be periodically retained or redesigned, which takes significant human and computational resources. This work presents the Online NeuroEvolution based Neural Architecture Search (ONE-NAS) algorithm, which to the authors' knowledge is the first neural architecture search algorithm capable of automatically designing and training new recurrent neural networks (RNNs) in an online setting. Without any pretraining, ONE-NAS utilizes populations of RNNs which are continuously updated with new network structures and weights in response to new multivariate input data. ONE-NAS is tested on real-world large-scale multivariate wind turbine data as well a univariate Dow Jones Industrial Average (DJIA) dataset, and is shown to outperform traditional statistical time series forecasting, including naive, moving average, and exponential smoothing methods, as well as state of the art online ARIMA strategies. CCS CONCEPTS• Theory of computation → Online algorithms; • Computing methodologies → Neural networks; • Applied computing → Forecasting.

show abstract

Improving neuroevolutionary transfer learning of deep recurrent neural networks through network-aware adaptation

Cited by 14 publications

References 10 publications

An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution

An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution

[Retracted] Optimization Algorithm of Tourism Security Early Warning Information System Based on Long Short‐Term Memory (LSTM)

ONE-NAS: An Online NeuroEvolution based Neural Architecture Search for Time Series Forecasting

Contact Info

Product

Resources

About