Q-learning for POMDP: An application to learning locomotion gaits

Abstract

This paper presents a Q-learning framework for learning optimal locomotion gaits in robotic systems modeled as coupled rigid bodies. Inspired by prevalence of periodic gaits in bio-locomotion, an open loop periodic input is assumed to (say) affect a nominal gait. The learning problem is to learn a new (modified) gait by using only partial noisy measurements of the state. The objective of learning is to maximize a given reward modeled as an objective function in optimal control settings. The proposed control architecture has three main components: (i) Phase modeling of dynamics by a single phase variable; (ii) A coupled oscillator feedback particle filter to represent the posterior distribution of the phase conditioned in the sensory measurements; and (iii) A Q-learning algorithm to learn the approximate optimal control law. The architecture is illustrated with the aid of a planar two-body system. The performance of the learning is demonstrated in a simulation environment.

Memory-based Deep Reinforcement Learning for POMDPs

“…Few works consider the more complex Partially Observable Markov Decision Process (POMDP) where the observation is just a partial representation of the underlying state. However, POMDP is ubiquitous in real robotics applications [16], [17], such as robot navigation [18], [19], robotic manipulation [20], autonomous driving [21], [22], [23], and planning under uncertainty [24], [25], [26], [27]. Partial observability may be due to limited sensing capability, or an incomplete system model resulting in uncertainty about full observability.…”

Bio-inspired Learning of Sensorimotor Control for Locomotion

“…This overall control system can be viewed as a central pattern generator (CPG) which integrates sensory information to learn closed-loop optimal control policies for biolocomotion. The framework presented here is based upon our prior research in [10] where phase reduction technique was introduced for a 2-link system and in [13] where the technique was extended to include learning for the 2-link system. The main contributions of this work over and above these prior publications are as follows:…”

“…1) The application involving the snake robot is new and more practically motivated than the simple 2-link model considered in [13].…”

“…3) A procedure to learn the sensor model is presented. This is in contrast to [10], [13], where the sensor model is assumed to be known. 4) The learning framework is numerically demonstrated in a simulation environment.…”