Q-Learning-Based Noise Covariance Adaptation in Kalman Filter for MARG Sensors Attitude Estimation

Abstract

The attitude estimation of a rigid body by magnetic, angular rate, and gravity (MARG) sensors is a research subject for a large variety of engineering applications. A standard solution for building up the observer is usually based on the Kalman filter and its different extensions for versatility and practical implementation. However, the performance of these observers has long suffered from the inaccurate process and measurement noise covariance matrices, which in turn entails tedious parameter turning procedures. To overcome the laborious noise covariance matrices regulation, we propose in this paper a Q-learning-based approach to autonomously adapt the values of process and measurement noise covariance matrices. The Qlearning method establishes a reinforcement learning mechanism that forces the noise covariance matrices pair with the least difference between predictions and measurements of output to be found in a predetermined candidate set of noise covariance matrices. The effectiveness of the Q-learning approach, applied to Extended Kalman filter-based attitude estimation, is validated through the Monte Carlo method that uses real flight data on an unmanned aerial vehicle.

A Dynamic Grid-based Q-learning for Noise Covariance Adaptation in EKF and its Application in Navigation

“…In this section, the Q-learning basics and their combination with the EKF are introduced, and then our previous work [10] on Q-learning-based adaptation is recalled.…”

“…In [10], we have proposed the QLEKF that runs three parallel EKFs at each time step: the traditional EKF, which sets some initial values of (Q, R) for all time steps and serves as the benchmark for Q-learning; the learning EKF, which searches appropriate noise covariance matrices from the grid by the Q-learning algorithm; and the learned EKF, which outputs the result of estimation according to the covariance matrices found by the learning EKF. Please refer to Alg.…”

“…2 in [10] for more details about the QLEKF. As shown in [10], the QLEKF exhibits the benefit of improving the EKF state estimation by searching for more appropriate noise covariance matrices from a predefined set of values. However, QLEKF contains potential insufficiencies:…”

“…Second, it is sufficient for Q-learning to explore and exploit all directions from the center element. Third, its all 9 elements can be visited whit a shorter learning period, compared to larger size grids in [8], [10]. Fourth, by dynamically updating the ratio r, q and the central element (Q c , R c ), it allows Q-learning to search in an arbitrarily large scope (by manipulating q, r, r and q) with an arbitrarily high precision (by setting r and q close to 1).…”

“…As one of the most important reinforcement learning methods, Q-learning [6] has drawn increasing interest in adapting the noise covariance matrices of EKF [7], [8], for its modelfree algorithm, low computation demand, and capability in achieving optimality in Markov decision processes [9]. In our previous work [10], a Q-learning-based EKF (QLEKF) is proposed to autonomously adapt the values of process and measurement noise covariance matrices in the attitude estimation of a rigid body. Though improvement is revealed in estimation errors compared to the traditional EKF using hand-tuned noise covariance matrices, the design of QLEKF involves certain amounts of heuristics and a rule of thumb.…”