Learning predictive state representations using non-blind policies

A central problem in artificial intelligence is to plan to maximize future reward under uncertainty in a partially observable environment. Models of such environments include Partially Observable Markov Decision Processes (POMDPs) [4] as well as their generalizations, Predictive State Representations (PSRs) [9] and Observable Operator Models (OOMs) [7]. POMDPs model the state of the world as a latent variable; in contrast, PSRs and OOMs represent state by tracking occurrence probabilities of a set of future events (called tests or characteristic events) conditioned on past events (called histories or indicative events). Unfortunately, exact planning algorithms such as value iteration [14] are intractable for most realistic POMDPs due to the curse of history and the curse of dimensionality [11]. However, PSRs and OOMs hold the promise of mitigating both of these curses: first, many successful approximate planning techniques designed to address these problems in POMDPs can easily be adapted to PSRs and OOMs [8,6]. Second, PSRs and OOMs are often more compact than their corresponding POMDPs (i.e., need fewer state dimensions), mitigating the curse of dimensionality. Finally, since tests and histories are observable quantities, it has been suggested that PSRs and OOMs should be easier to learn than POMDPs; with a successful learning algorithm, we can look for a model which ignores all but the most important components of state, reducing dimensionality still further.In this paper we take an important step toward realizing the above hopes. In particular, we propose and demonstrate a fast and statistically consistent spectral algorithm which learns the parameters of a PSR directly from sequences of action-observation pairs. We then close the loop from observations to actions by planning in the learned model and recovering a policy which is near-optimal in the original environment. Closing the loop is a much more stringent test than simply checking short-term prediction accuracy, since the quality of an optimized policy depends strongly on the accuracy of the model: inaccurate models typically lead to useless plans. Categories and Subject Descriptors General TermsAlgorithms, Theory KeywordsMachine Learning, Predictive State Representations, Planning Closing the Learning-Planning Loop with PSRsWe propose a novel algorithm for learning a variant of PSRs [12] directly from execution traces. Our algorithm is closely related to subspace identification for linear dynamical systems (LDSs) [15] and spectral algorithms for Hidden Markov Models (HMMs) [5] and reduced-rank HMMs [13]. We then use the algorithm to learn a model of a simulated high-dimensional, vision-based mobile robot planning task, and compute a policy by approximate point-based planning in the learned model [6]. Finally, we show that the learned state space compactly captures the essential features of the environment, allows accurate prediction, and enables successful and efficient planning.By comparison, previous POMDP learners such as ExpectationMaximi...

show abstract

“…[16,3]) and OOMs (e.g. [10]), none have been shown to learn models that are accurate enough for lookahead planning.…”

Section: Closing the Learning-planning Loop With Psrsmentioning

confidence: 99%

Closing the Learning-Planning Loop with Predictive State Representations

Boots¹,

Siddiqi²,

Gordon³

2010

Robotics: Science and Systems VI

153

View full text Add to dashboard Cite

show abstract

“…We select an action a j+1 according to this distribution, we execute this action, and then we use this new information (o j+1 , a j+1 ) to update our core test probabilities by using Equation 4. We add the event (o j+1 , a j+1 ) at the end of the history h j , which becomes h j+1 , and we repeat the same process for the next steps.…”

Section: Predictive Policy Representations (Pprs)mentioning

confidence: 99%

“…if the column of oaq inP is linearly dependent on 4 the columns of the core tests q ∈ Q; then 5 Find the vector m oaq by solving the linear…”

Section: Learning Pprs Vs Learning Fscsmentioning

confidence: 99%

A Predictive Model for Imitation Learning in Partially Observable Environments

Boularias

2008

2008 Seventh International Conference on Machine Learning and Applications

View full text Add to dashboard Cite

Learning by imitation has shown to be a powerful paradigm for automated learning in autonomous robots. This paper presents a general framework of learning by imitation for stochastic and partially observable systems. The model is a Predictive Policy Representation (PPR) whose goal is to represent the teacher‘s policies without any reference to states. The model is fully described in terms of actions and observations only. We show how this model can efficiently learn the personal behavior and preferences of an assistive robot user

show abstract

“…Prior work on learning predictive models have mostly taken the approach of learning a complete model of the system with the objective of obtaining good predictive accuracy for all possible future behaviors, then planning with the learned model. Thus they either assume the training data is acquired through interaction with the environment using a purely exploratory policy [4], or else side-step the exploration problem by learning from a batch of pre-sampled action-observation trajectories [2].…”

Section: Introductionmentioning

confidence: 99%

Goal-Directed Online Learning of Predictive Models

Ong

Grinberg

Pineau

2012

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. We present an algorithmic approach for integrated learning and planning in predictive representations. The approach extends earlier work on predictive state representations to the case of online exploration, by allowing exploration of the domain to proceed in a goal-directed fashion and thus be more efficient. Our algorithm interleaves online learning of the models, with estimation of the value function. The framework is applicable to a variety of important learning problems, including scenarios such as apprenticeship learning, model customization, and decisionmaking in non-stationary domains.

show abstract

Learning predictive state representations using non-blind policies

Cited by 21 publications

References 6 publications

Closing the Learning-Planning Loop with Predictive State Representations

Closing the Learning-Planning Loop with Predictive State Representations

A Predictive Model for Imitation Learning in Partially Observable Environments

Goal-Directed Online Learning of Predictive Models

Contact Info

Product

Resources

About