Extending the Applicability of POMDP Solutions to Robotic Tasks

Abstract

Abstract-Partially-Observable Markov Decision Processes (POMDPs) are used in many robotic task classes from soccer to household chores. Determining an approximately optimal action policy for POMDPs is PSPACE-complete, and the exponential growth of computation time prohibits solving large tasks. This paper describes two techniques to extend the range of robotic tasks that can be solved using a POMDP. Our first technique reduces the motion constraints of a robot, and then uses state-of-the-art robotic motion planning techniques to respect the true motion constraints at runtime. We then propose a novel task decomposition that can be applied to some indoor robotic tasks. This decomposition transforms a long time horizon task into a set of shorter tasks.We empirically demonstrate the performance gain provided by these two techniques through simulated execution in a variety of environments. Comparing a direct formulation of a POMDP to solving our proposed reductions, we conclude that the techniques proposed in this paper can provide significant enhancement to current POMDP solution techniques, extending the POMDP instances that can be solved to include large, continuous-state robotic tasks.

Reinforcement Learning Based Hierarchical Multi-Agent Robotic Search Team in Uncertain Environment

“…The size of input POMDP is decreased leading to an enhanced runtime for POMDP solver. Output policy of the solver is re-evaluated to make sure it works for every input problem [13].…”

“…All of the aforementioned techniques are related to multi-agent systems but some do not present a viable method for agents to learn during exploration [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17], also agents which do learning themselves cannot detect errors on their own and hence those systems are not fault tolerant [18][19][20][21][22][23][24]. For exploration, agents must also be equipped with making decisions in less time, so there should be limited computation time for each agent.…”

Learning to calibrate: Reinforcement learning for guided calibration of visual–inertial rigs

“…MDPs and partially-observable Markov decision processes (POMDPs) have been used extensively in the context of robotics (Grady et al, 2015). Winterer et al (2017) used POMDPs for motion planning and Chatterjee et al (2015) used POMDPs for qualitative analysis; however, to the best of the authors’ knowledge, the sensor calibration problem has not been cast as a MDP or POMDP.…”

Energy expenditure estimation through daily activity recognition using a smart-phone

“…From trials and errors, the agent will build a predictive model. Traditionnaly, the POMDP framework is used in robotic tasks [15]. Processing its sensors' signals, the robot builds a representation of the environment from which it can take actions.…”