Solving partially observable problems with inaccurate PSR models

Liu, Yunlong; Yang, Zijiang; Ji, Guoli

doi:10.1016/j.ins.2014.06.034

Cited by 6 publications

(4 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then the data is translated into the form of action-observation -p(T |•) sequences. For example, a sequence end for 14: end for 15: Output: T (2011); Liu et al (2014). Subsequently, we compute the state-transition functions in the transformed data and build the MDP model.…”

Section: Basis Selection Via Model Entropymentioning

confidence: 99%

See 1 more Smart Citation

Selecting Bases in Spectral learning of Predictive State Representations via Model Entropy

Liu,

Zhu

2016

Preprint

Self Cite

View full text Add to dashboard Cite

Predictive State Representations (PSRs) are powerful techniques for modelling dynamical systems, which represent a state as a vector of predictions about future observable events (tests). In PSRs, one of the fundamental problems is the learning of the PSR model of the underlying system. Recently, spectral methods have been successfully used to address this issue by treating the learning problem as the task of computing an singular value decomposition (SVD) over a submatrix of a special type of matrix called the Hankel matrix. Under the assumptions that the rows and columns of the submatrix of the Hankel Matrix are sufficient (which usually means a very large number of rows and columns, and almost fails in practice) and the entries of the matrix can be estimated accurately, it has been proven that the spectral approach for learning PSRs is statistically consistent and the learned parameters can converge to the true parameters. However, in practice, due to the limit of the computation ability, only a finite set of rows or columns can be chosen to be used for the spectral learning. While different sets of columns usually lead to variant accuracy of the learned model, in this paper, we propose an approach for selecting the set of columns, namely basis selection, by adopting a concept of model entropy to measure the accuracy of the learned model. Experimental results are shown to demonstrate the effectiveness of the proposed approach.

show abstract

Section: Basis Selection Via Model Entropymentioning

confidence: 99%

“…There are two main problems in PSRs. One is the learning of the PSR model; the other is the application of the learned model, including predicting and planning Rosencrantz Liu Zhu et al (2004); Liu et al (2014). The state-of-the-art technique for addressing the learning problem is the spectral approach Boots et al (2011).…”

Section: Introductionmentioning

confidence: 99%

Selecting Bases in Spectral learning of Predictive State Representations via Model Entropy

Liu,

Zhu

2016

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Some of the work aims for learning the complete model of the underlying system. Huang et al [4][5][6] propose the planning methods based on the PSR model. Song et al [7] and Somani et al [8] propose the planning method based on the POMDP model.…”

Section: Introductionmentioning

confidence: 99%

Deep Q-Network with Predictive State Models in Partially Observable Domains

Liu

2020

Mathematical Problems in Engineering

Self Cite

View full text Add to dashboard Cite

While deep reinforcement learning (DRL) has achieved great success in some large domains, most of the related algorithms assume that the state of the underlying system is fully observable. However, many real-world problems are actually partially observable. For systems with continuous observation, most of the related algorithms, e.g., the deep Q-network (DQN) and deep recurrent Q-network (DRQN), use history observations to represent states; however, they often make computation-expensive and ignore the information of actions. Predictive state representations (PSRs) can offer a powerful framework for modelling partially observable dynamical systems with discrete or continuous state space, which represents the latent state using completely observable actions and observations. In this paper, we present a PSR model-based DQN approach which combines the strengths of the PSR model and DQN planning. We use a recurrent network to establish the recurrent PSR model, which can fully learn dynamics of the partially continuous observable environment. Then, the model is used for the state representation and update of DQN, which makes DQN no longer rely on a fixed number of history observations or recurrent neural network (RNN) to represent states in the case of partially observable environments. The strong performance of the proposed approach is demonstrated on a set of robotic control tasks from OpenAI Gym by comparing with the technique with the memory-based DRQN and the state-of-the-art recurrent predictive state policy (RPSP) networks. Source code is available at https://github.com/RPSR-DQN/paper-code.git.

show abstract

“…One commonly used technique for solving such partially observable problems is to model the dynamics of the environments firstly, for example, the Partially Observable Markov Decision Processes (POMDP) (Kaelbling, Littman, & Cassandra, 1998;Ross, Pineau, Paquet, & Chaib-Draa, 2008) and Predictive State Representations (PSRs) (Littman, Sutton, & Singh, 2001;Liu, Tang, & Zeng, 2015;Liu, Zhu, Zeng, & Dai, 2016;Talvitie & Singh, 2011) approach, and then the problem can be solved using the obtained model. Although POMDPs and PSRs provide general frameworks to solve partially observable problems, they rely heavily on a known and accurate model of the environment (Liu, Yang, & Ji, 2014;Spaan & Vlassis, 2005;Pineau, Gordon, & Thrun, 2006;Ye, Somani, Hsu, & Lee, 2017). However, in real-world applications it is extremely difficult to build an accurate model.…”

Section: Introductionmentioning

confidence: 99%

Combining Offline Models and Online Monte-Carlo Tree Search for Planning from Scratch

Liu,

Zheng

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Planning in stochastic and partially observable environments is a central issue in artificial intelligence. One commonly used technique for solving such a problem is by constructing an accurate model firstly. Although some recent approaches have been proposed for learning optimal behaviour under model uncertainty, prior knowledge about the environment is still needed to guarantee the performance of the proposed algorithms. With the benefits of the Predictive State Representations (PSRs) approach for state representation and model prediction, in this paper, we introduce an approach for planning from scratch, where an offline PSR model is firstly learned and then combined with online Monte-Carlo tree search for planning with model uncertainty. By comparing with the state-of-the-art approach of planning with model uncertainty, we demonstrated the effectiveness of the proposed approaches along with the proof of their convergence. The effectiveness and scalability of our proposed approach are also tested on the RockSample problem, which are infeasible for the state-of-the-art BA-POMDP based approaches.

show abstract

Solving partially observable problems with inaccurate PSR models

Cited by 6 publications

References 26 publications

Selecting Bases in Spectral learning of Predictive State Representations via Model Entropy

Selecting Bases in Spectral learning of Predictive State Representations via Model Entropy

Deep Q-Network with Predictive State Models in Partially Observable Domains

Combining Offline Models and Online Monte-Carlo Tree Search for Planning from Scratch

Contact Info

Product

Resources

About