An online and approximate solver for POMDPs with continuous action space

Seiler, Konstantin M.; Kurniawati, Hanna; Singh, Surya P. N.

doi:10.1109/icra.2015.7139503

Cited by 70 publications

(53 citation statements)

References 17 publications

(28 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…II. RELATED WORK POMDP planning has a huge literature (see, e.g., [8], [13], [25], [23], [13], [22], [24], [21] ). Our brief review focuses on online search algorithms.…”

mentioning

confidence: 99%

POMDP-lite for robust robot planning under uncertainty

Chen

Hsu

Lee

2016

2016 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Abstract-The partially observable Markov decision process (POMDP) provides a principled general model for planning under uncertainty. However, solving a general POMDP is computationally intractable in the worst case. This paper introduces POMDP-lite, a subclass of POMDPs in which the hidden state variables are constant or only change deterministically. We show that a POMDP-lite is equivalent to a set of fully observable Markov decision processes indexed by a hidden parameter and is useful for modeling a variety of interesting robotic tasks. We develop a simple model-based Bayesian reinforcement learning algorithm to solve POMDP-lite models. The algorithm performs well on large-scale POMDP-lite models with up to 10 20 states and outperforms the state-of-the-art general-purpose POMDP algorithms. We further show that the algorithm is near-Bayesian-optimal under suitable conditions.

show abstract

“…II. RELATED WORK POMDP planning has a huge literature (see, e.g., [8], [13], [25], [23], [13], [22], [24], [21] ). Our brief review focuses on online search algorithms.…”

mentioning

confidence: 99%

POMDP-lite for robust robot planning under uncertainty

Chen

Hsu

Lee

2016

2016 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

show abstract

“…When the action space is large, such enumeration is no longer feasible. Several solvers [27]- [29] have even been proposed for continuous action spaces under some technical conditions. However, to the best of our knowledge, there is no general POMDP solver for large discrete action spaces.…”

Section: Pomdp Formulationmentioning

confidence: 99%

Sequential multi-class labeling in crowdsourcing

Kang

Tay

2017

Proceedings of the International Conference on Web Intelligence

View full text Add to dashboard Cite

Abstract-We consider a crowdsourcing platform where workers' responses to questions posed by a crowdsourcer are used to determine the hidden state of a multi-class labeling problem. As workers may be unreliable, we propose to perform sequential questioning in which the questions posed to the workers are designed based on previous questions and answers. We propose a Partially-Observable Markov Decision Process (POMDP) framework to determine the best questioning strategy, subject to the crowdsourcer's budget constraint. As this POMDP formulation is in general intractable, we develop a suboptimal approach based on a q-ary Ulam-Rényi game. We also propose a sampling heuristic, which can be used in tandem with standard POMDP solvers, using our Ulam-Rényi strategy. We demonstrate through simulations that our approaches outperform a non-sequential strategy based on error correction coding and which does not utilize workers' previous responses.

show abstract

“…While POMCP, ABT and DESPOT can handle continuous state spaces, they are limited to discrete action spaces and discrete or discretized observation spaces. Very recently, on-line solvers have been proposed that can handle continuous action [15,46] and observation spaces [46,47]. Nevertheless, efficiently solving POMDP problems with continuous action spaces remains an open problem.…”

Section: Pomdp Solversmentioning

confidence: 99%

“…Moreover, the curse of history is amplified by the large action space imposed by a 6DOFs manipulator. Methods have been proposed to alleviate this issue [15,46]. However, existing solvers can only perform well for problems with 3-4 continuous action spaces [15], while a method that can perform well for problems with 100,000 discrete actions was only recently proposed [98].…”

Section: Candyscoopermentioning

confidence: 99%

“…The underlying principle of these methods is the fact that in many applications a near-optimal strategy is "good enough" to accomplish a given task. Several general approximate POMDP solvers -that do not restrict the type of dynamics and sensing model of the system, nor the type of distributions used to represent uncertainty-can now compute good motion strategies on-line with a 1-10Hz update rate for a number of robotic problems [12][13][14][15]. Despite these advances, applying state-of-the-art approximate POMDP solvers to realistic robotic systems remains challenging.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Tractable POMDP-planning for robots with complex non-linear dynamics

Hoerger¹

View full text Add to dashboard Cite

Planning under partial observability is an essential capability of autonomous robots. While robots operate in the real world, they are inherently subject to various uncertainties such a control and sensing errors, and limited information regarding the operating environment. Conceptually these type of planning problems can be solved in a principled manner when framed as a Partially Observable Markov Decision Process (POMDP). POMDPs model the aforementioned uncertainties as conditional probability functions and estimate the state of the system as probability functions over the state space, called beliefs. Instead of computing the best strategy with respect to single states, POMDP solvers compute the best strategy with respect to beliefs. Solving a POMDP exactly is computationally intractable in general.However, in the past two decades we have seen tremendous progress in the development of approximately optimal solvers that trade optimality for computational tractability. Despite this progress, approximately solving POMDPs for systems with complex non-linear dynamics remains challenging.Most state-of-the-art solvers rely on a large number of expensive forward simulations of the system to find an approximate-optimal strategy. For systems with complex non-linear dynamics that admit no closed-form solution, this strategy can become prohibitively expensive. Another difficulty in applying POMDPs to physical robots with complex transition dynamics is the fact that almost all implementations of state-of-the-art on-line POMDP solvers restrict the user to specific data structures for the POMDP model, and the model has to be hard-coded within the solver implementation. This, in turn, severely hinders the process of applying POMDPs to physical robots.In this thesis we aim to make POMDPs more practical for realistic robotic motion planning tasks under partial observability. We show that systematic approximations of complex, non-linear transition dynamics can be used to design on-line POMDP solvers that are more efficient than current solvers. Furthermore, we propose a new software-framework that supports the user in modeling complex planning problems under uncertainty with minimal implementation effort.

show abstract

An online and approximate solver for POMDPs with continuous action space

Cited by 70 publications

References 17 publications

POMDP-lite for robust robot planning under uncertainty

POMDP-lite for robust robot planning under uncertainty

Sequential multi-class labeling in crowdsourcing

Tractable POMDP-planning for robots with complex non-linear dynamics

Contact Info

Product

Resources

About