An Improved Q-learning Approach with Kalman Filter for Self-balancing Robot Using OpenAI

Abstract

A two-wheeled self-balancing robot (SBR) is a typical example in control systems that works on the principle of an inverted pendulum. In this paper, we experiment to see how the learning and stability performance varies based on Kalman filter introduction for IMU noise filtering and controlling the robot using reinforcement learning. All the implementation is performed in ROS and Gazebo, and Q-learning is implemented using OpenAI (toolkit for development of Reinforcement learning) for ROS, i.e., Openai_ros package. Our work deals with a novel approach of providing the angular output from IMU to Kalman filter and passing it to the input of Q-learning for balancing control. Finally, we analyze the results with and without using Kalman filter from the output of IMU before passing it to Q-learning and evaluate the performance based on robot's learning behavior and its robustness.

Deep‐Q‐Network hybridization with extended Kalman filter for accelerate learning in autonomous navigation with auxiliary security module

“…Recently, the authors in Reference 18 published research involving the use of KF to improve reinforcement learning applied to the control of a two‐wheeled robot, similar to the inverted pendulum problem. Different from our proposal, which uses EKF as a predictor of future robot states, the authors apply KF to filter out noisy information in sensory data before being used as a state in Q‐Learning.…”

“…This technique has been shown to improve the accuracy of states measured by a sensor, in this case the IMU (inertial measurement unit), thus it increases the stabilization of the obtained rewards and the transient response of the control system. Therefore, from the results presented in Reference 18, we are led to investigate if in addition to providing greater stabilization, KF can accelerate a newer or current reinforcement learning technique such as DQN.…”

Research on Deep Q-Network Hybridization with Extended Kalman Filter in Maneuvering Decision of Unmanned Combat Aerial Vehicles

“…The true values of the weight parameters of the deep Q-network may have some deviation or fluctuation from the values we train, which may come from factors such as noise in the training data, randomness of the optimization algorithm, and complexity of the network structure. The uncertainty of network parameters can affect the performance and stability of the network, so it is necessary to analyze and filter out the uncertainty of network parameters [16].…”

Deep-Q-Network Hybridization with Extended Kalman Filter for Accelerate Learning in Autonomous Navigation with the Auxiliary Security Module

“…Recently, the authors in 16 published research involving the use of KF to improve reinforcement learning applied to the control of a two-wheeled robot, similar to the inverted pendulum problem. Different from our proposal, which uses EKF as a predictor of future robot states, the authors apply KF to filter out noisy information in sensory data before being used as a state in Q-Learning.…”

“…This technique has been shown to improve the accuracy of states measured by a sensor, in this case the IMU (Inertial Measurement Unit), thus it increases the stabilization of the obtained rewards and the transient response of the control system. Therefore, from the results presented in 16 , we are led to investigate if in addition to providing greater stabilization, KF can accelerate a newer or current reinforcement learning technique such as DQN.…”