Closing the Learning-Planning Loop with Predictive State Representations

Abstract

A central problem in artificial intelligence is to plan to maximize future reward under uncertainty in a partially observable environment. Models of such environments include Partially Observable Markov Decision Processes (POMDPs) [4] as well as their generalizations, Predictive State Representations (PSRs) [9] and Observable Operator Models (OOMs) [7]. POMDPs model the state of the world as a latent variable; in contrast, PSRs and OOMs represent state by tracking occurrence probabilities of a set of future events (called tests or characteristic events) conditioned on past events (called histories or indicative events). Unfortunately, exact planning algorithms such as value iteration [14] are intractable for most realistic POMDPs due to the curse of history and the curse of dimensionality [11]. However, PSRs and OOMs hold the promise of mitigating both of these curses: first, many successful approximate planning techniques designed to address these problems in POMDPs can easily be adapted to PSRs and OOMs [8,6]. Second, PSRs and OOMs are often more compact than their corresponding POMDPs (i.e., need fewer state dimensions), mitigating the curse of dimensionality. Finally, since tests and histories are observable quantities, it has been suggested that PSRs and OOMs should be easier to learn than POMDPs; with a successful learning algorithm, we can look for a model which ignores all but the most important components of state, reducing dimensionality still further.In this paper we take an important step toward realizing the above hopes. In particular, we propose and demonstrate a fast and statistically consistent spectral algorithm which learns the parameters of a PSR directly from sequences of action-observation pairs. We then close the loop from observations to actions by planning in the learned model and recovering a policy which is near-optimal in the original environment. Closing the loop is a much more stringent test than simply checking short-term prediction accuracy, since the quality of an optimized policy depends strongly on the accuracy of the model: inaccurate models typically lead to useless plans. Categories and Subject Descriptors General TermsAlgorithms, Theory KeywordsMachine Learning, Predictive State Representations, Planning Closing the Learning-Planning Loop with PSRsWe propose a novel algorithm for learning a variant of PSRs [12] directly from execution traces. Our algorithm is closely related to subspace identification for linear dynamical systems (LDSs) [15] and spectral algorithms for Hidden Markov Models (HMMs) [5] and reduced-rank HMMs [13]. We then use the algorithm to learn a model of a simulated high-dimensional, vision-based mobile robot planning task, and compute a policy by approximate point-based planning in the learned model [6]. Finally, we show that the learned state space compactly captures the essential features of the environment, allows accurate prediction, and enables successful and efficient planning.By comparison, previous POMDP learners such as ExpectationMaximi...

Model learning for robot control: a survey

“…Thus, learning predictive state distribution incorporates the generation of a set of tests and the evaluations of these test samples. On a high level, learning how to generate and evaluate test samples can be can be seen as a model learning problem, where two models (i.e., for test generation and evaluation) need to be approximated (Boots et al 2010). Formulated in a probabilistic framework, these probabilistic models can be estimated empirically from samples (Boots et al 2010).…”

“…In the first case, learning algorithms have to deal with massive amounts of data, such as in learning inverse dynamics. In this scenario, the algorithms need to be efficient in terms of computation without sacrificing the learning accuracy (Bottou et al 2007). In the second scenario, there is only few data available for learning, as the data generation may be too tedious and expensive.…”

Learning predictive models of a depth camera & manipulator from raw execution traces

“…Popular latent variable models of stochastic processes are often learned using heuristics such as Expectation Maximization (EM), which suffer from bad local optima and slow convergence rates. Recent PSR learning algorithms rely on spectral methods [10], [27] and kernel methods [11] which are statistically consistent.…”

“…This problem is far more difficult than the simulated problems explored in previous PSR work [10], [12]- [14]. Additionally, the manipulator used in our experiments has many additional degrees of freedom compared to the systems considered in recent work on bootstrapping in robotics [15]- [17].…”

“…We approach this difficult problem from a machine learning perspective. We dispense with problem-dependent geometric and physical intuitions and instead model the sensorimotor data as a Predictive State Representation (PSR) [6], [7], a general probabilistic modeling framework that can represent a wide variety of stochastic process models including Kalman filters (KFs) [8], input output hidden Markov models (IO-HMMs) [9], [10], and nonparametric dynamical system models [11].…”