Online Training of LSTM Networks in Distributed Systems for Variable Length Data Sequences

Abstract

In this brief, we investigate online training of long short term memory (LSTM) architectures in a distributed network of nodes, where each node employs an LSTM-based structure for online regression. In particular, each node sequentially receives a variable length data sequence with its label and can only exchange information with its neighbors to train the LSTM architecture. We first provide a generic LSTM-based regression structure for each node. In order to train this structure, we put the LSTM equations in a nonlinear state-space form for each node and then introduce a highly effective and efficient distributed particle filtering (DPF)-based training algorithm. We also introduce a distributed extended Kalman filtering-based training algorithm for comparison. Here, our DPF-based training algorithm guarantees convergence to the performance of the optimal LSTM coefficients in the mean square error sense under certain conditions. We achieve this performance with communication and computational complexity in the order of the first-order gradient-based methods. Through both simulated and real-life examples, we illustrate significant performance improvements with respect to the state-of-the-art methods.

Enhanced Accuracy of Heart Disease Prediction using Machine Learning and Recurrent Neural Networks Ensemble Majority Voting Method

“…In RNN section, LSTM consider as the determination with four important factors (forget gate (f g), input gate (I g), output gate (O g) and cell state) have an ample usage for the image analysis along with text and audio signal analysis but is extensively usage in time series analysis, transcribed analysis, voice recognition and health testimony [22]. The major detriment of the RNN model was vanishing gradient problem, LSTM increased the input and output capability of RNN to solve these issues and it uses logical memory to learn sequence vector.…”

Adaptive fusion localization mechanism towards TDoA and IMU data with LSTM correction Method

“…The LSTM network is a time recurrent neural network that is an enhancement to the RNN neural network and is suitable for processing and predicting time series [13][14][15][16] . In this paper, the LSTM network model is used to modify the carrier trajectory.…”

“…However, the carrier motion trajectory in the time period close to the current time is highly correlated with the trajectory of the carrier at the next time, and thus the partial information remains. The forgetting gate output expression is as shown in equation (15). 1 ( )…”

Recurrent neural networks based online learning algorithms for distributed systems

“…Toplu egitimde tüm veri ulaşılabilir durumdadır ve birlikte işlenir. Ancak, büyük veri uygulamalarında, tüm verinin aynı yerde tutulması çeşitli depolama sorunlarına neden olmaktadır [4]. Ek olarak, birçok uygulamada veriler ardışık olarak elde edilmektedir ve bu durum toplu egitimin kullanılmasını önlemektedir.…”