Robust Finite-State Controllers for Uncertain POMDPs

Cubuktepe, Murat; Jansen, Nils; Junges, Sebastian; Marandi, Ahmadreza; Suilen, Marnix; Topcu, Ufuk

doi:10.1609/aaai.v35i13.17401

Cited by 16 publications

(15 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A policy represented by an FSC can get arbitrarily close to the optimal policy as κ grows (Bonet 2002). Computing finite-state controllers that aim to maximize the expected (discounted) reward can be done in several ways, such as gradient descent (Meuleau et al 1999b) and convex optimization (Amato, Bernstein, and Zilberstein 2010;Junges et al 2018;Cubuktepe et al 2021).…”

Section: Mdps Pomdps and Fscsmentioning

confidence: 99%

Safe Policy Improvement for POMDPs via Finite-State Controllers

Simão

Suilen

Jansen

2023

AAAI

View full text Add to dashboard Cite

We study safe policy improvement (SPI) for partially observable Markov decision processes (POMDPs). SPI is an offline reinforcement learning (RL) problem that assumes access to (1) historical data about an environment, and (2) the so-called behavior policy that previously generated this data by interacting with the environment. SPI methods neither require access to a model nor the environment itself, and aim to reliably improve upon the behavior policy in an offline manner. Existing methods make the strong assumption that the environment is fully observable. In our novel approach to the SPI problem for POMDPs, we assume that a finite-state controller (FSC) represents the behavior policy and that finite memory is sufficient to derive optimal policies. This assumption allows us to map the POMDP to a finite-state fully observable MDP, the history MDP. We estimate this MDP by combining the historical data and the memory of the FSC, and compute an improved policy using an off-the-shelf SPI algorithm. The underlying SPI method constrains the policy space according to the available data, such that the newly computed policy only differs from the behavior policy when sufficient data is available. We show that this new policy, converted into a new FSC for the (unknown) POMDP, outperforms the behavior policy with high probability. Experimental results on several well-established benchmarks show the applicability of the approach, even in cases where finite memory is not sufficient.

show abstract

Section: Mdps Pomdps and Fscsmentioning

confidence: 99%

Safe Policy Improvement for POMDPs via Finite-State Controllers

Simão

Suilen

Jansen

2023

AAAI

View full text Add to dashboard Cite

show abstract

“…For a detailed handling of different types of uncertainty sets, we refer to [83]. Extensions to uncertain POMDPs also exist [31,79]. Distributions over Parameters.…”

Section: Epiloguementioning

confidence: 99%

Parameter Synthesis in Markov Models: A Gentle Survey

Jansen¹,

Junges²,

Katoen³

2022

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

This paper surveys the analysis of parametric Markov models whose transitions are labelled with functions over a finite set of parameters. These models are symbolic representations of uncountable many concrete probabilistic models, each obtained by instantiating the parameters. We consider various analysis problems for a given logical specification ϕ: do all parameter instantiations within a given region of parameter values satisfy ϕ?, which instantiations satisfy ϕ and which ones do not?, and how can all such instantiations be characterised, either exactly or approximately? We address theoretical complexity results and describe the main ideas underlying state-of-the-art algorithms that established an impressive leap over the last decade enabling the fully automated analysis of models with millions of states and thousands of parameters.

show abstract

“…The methods presented in [13,22] exploit a repetitive structure in parametric MCs to accelerate the construction of closed form solutions and are not applicable to MDPs. Parametric models have been used to support the design of systems [2,8] or their adaption [6,9], to find policies for partially observable systems [11], to analyse Bayesian networks [34], and to speed up the analysis of, e.g., software product lines [10,37]. On top of technical differences, none of these approaches uses a hierarchical decomposition of an MDP or uses the results of the analysis in the analysis of a larger MDP.…”

Section: Related Workmentioning

confidence: 99%

Abstraction-Refinement for Hierarchical Probabilistic Models

Junges

Spaan

2022

Computer Aided Verification

Self Cite

View full text Add to dashboard Cite

Markov decision processes are a ubiquitous formalism for modelling systems with non-deterministic and probabilistic behavior. Verification of these models is subject to the famous state space explosion problem. We alleviate this problem by exploiting a hierarchical structure with repetitive parts. This structure not only occurs naturally in robotics, but also in probabilistic programs describing, e.g., network protocols. Such programs often repeatedly call a subroutine with similar behavior. In this paper, we focus on a local case, in which the subroutines have a limited effect on the overall system state. The key ideas to accelerate analysis of such programs are (1) to treat the behavior of the subroutine as uncertain and only remove this uncertainty by a detailed analysis if needed, and (2) to abstract similar subroutines into a parametric template, and then analyse this template. These two ideas are embedded into an abstraction-refinement loop that analyses hierarchical MDPs. A prototypical implementation shows the efficacy of the approach.

show abstract

Robust Finite-State Controllers for Uncertain POMDPs

Cited by 16 publications

References 27 publications

Safe Policy Improvement for POMDPs via Finite-State Controllers

Safe Policy Improvement for POMDPs via Finite-State Controllers

Parameter Synthesis in Markov Models: A Gentle Survey

Abstraction-Refinement for Hierarchical Probabilistic Models

Contact Info

Product

Resources

About