Extended Kalman filter neural network training: experimental results and algorithm improvements

Abstract

It is well known that the Extended Kalman Filter (EKF) neural network training algorithm is superior to the standard backpropagation algorithm.However. there are many variations on the EKF implementation that can significantly effect its performance. For example, improper initialization of three parameters cause the algorithm to perform poorly. There are also two advanced methods, de-coupling and multistreaming, which need to be properly applied based on the specifics of the problem. This paper presents the results of extensive experimentation in applying the EKF training method for recurrent and static neural networks. The goal is to demonstrate how different variations on its implementation effect performance and to find methods to optimize performance. The paper examines the effects of decoupling, multi-streaming, and initial values of constants used by the algorithm. Three new ideas are suggested that can lead to improved performance.These ideas are: initializing parameters to values outside the range previously suggested, a new decoupling strategy, and reducing the update rate of the error covariance matrix for faster training.

Large-Scale IP Traffic Matrix Estimation Based on the Recurrent Multilayer Perceptron Network

“…And during the training of RMLP network, the recent phenomenon [10], [11] often occurs. So, during the training of the multi-input and multi-output modified RMLP network model, the decoupled extended Kalman filter (DEKF) method [10], [11], [12], multi-stream procedure and truncated backpropagation through time (BPTT) [10], [11] are used to accelerate the convergence and improve the training performance. For the training process of our model, Algorithm 2.1 is presented as follows.…”

Fault detection of an engine using a neural network trained by the smooth variable structure filter

“…Most of these techniques like quasi-Newton and Levenburg-Marquardt demonstrate better performance as they involve second-order derivative information. In addition, these algorithms are implemented in a batch (multistreaming) mode where weights are updated based on more than one training sample in the training set, in contrast with the conventional BP where weights are updated by involving only one training sample (a serial mode) [3]. Even though second-order algorithms have proven to outperform the classical first-order BP, they may suffer from poor convergence properties due to problems with local minima [2].…”

“…It involves training M multiple identical parallel neural networks using several training samples followed by a single weight update using the entire M networks errors. The above algorithm can be adjusted to multi-streaming mode by replacing m with M x m [3].…”

Neural Networks Trained with Levenberg-Marquardt-Iterated Extended Kalman Filter for Mobile Robot Trajectory Tracking

“…This is because it is easy to implement and exhibit computationally efficient calculation which is especially useful for nonlinear systems and practical applications see [14]. There are many variables that affect EKF training algorithm performances.…”

“…These variables are matrices that must be correctly initialized otherwise the EKF training algorithm can exhibit poor performance. These matrices are the estimation error covariance matrix (P), the measurement covariance matrix (R), and the additional process noise matrix (Q) [14]. the iterative version of EKF is the iterative extended Kalman filter (IEKF), which improves the linearization of the extended Kalman filter by recursively, this version is powerful than the standard EKF for neural network training [15].…”