The Optimal Control of Partially Observable Markov Processes over a Finite Horizon

Smallwood, Richard D.; Sondik, Edward J.

doi:10.1287/opre.21.5.1071

Cited by 1,091 publications

(732 citation statements)

References 4 publications

Supporting

Mentioning

708

Contrasting

Unclassified

Order By: Relevance

“…24 and demonstrate the empirical performance of the model clustering approach on two problem domains: the multiagent tiger problem (tiger's location resets if a door is opened) and a multiagent version of the machine maintenance problem [34], both of which are described in the Appendix. In particular, we show that the quality of the policies generated using our method approaches that of the exact policy as K increases.…”

Section: Resultsmentioning

confidence: 99%

“…They generalize POMDPs [19,34] to multiagent settings by including other agents' computable models in the state space along with the states of the physical environment. The models encompass all information inf uencing the agents' behaviors, including their preferences, capabilities, and beliefs, and are thus analogous to types in Bayesian games as f rst envisioned by Harsanyi [17].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Graphical models for interactive POMDPs: representations and solutions

Doshi

Zeng

Chen

2008

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

We develop new graphical representations for the problem of sequential decision making in partially observable multiagent environments, as formalized by interactive partially observable Markov decision processes (I-POMDPs). The graphical models called interactive inf uence diagrams (I-IDs) and their dynamic counterparts, interactive dynamic inf uence diagrams (I-DIDs), seek to explicitly model the structure that is often present in real-world problems by decomposing the situation into chance and decision variables, and the dependencies between the variables. I-DIDs generalize DIDs, which may be viewed as graphical representations of POMDPs, to multiagent settings in the same way that IPOMDPs generalize POMDPs. I-DIDs may be used to compute the policy of an agent given its belief as the agent acts and observes in a setting that is populated by other interacting agents. Using several examples, we show how I-IDs and I-DIDs may be applied and demonstrate their usefulness. We also show how the models may be solved using the standard algorithms that are applicable to DIDs.Solving I-DIDs exactly involves knowing the solutions of possible models of the other agents. The space of models grows exponentially with the number of time steps. We present a method of solving I-DIDs approximately by limiting the number of other agents' candidate models at each time step to a constant. We do this by clustering models that are likely to be behaviorally equivalent and selecting a representative set from the clusters. We discuss the error bound of the approximation technique and demonstrate its empirical performance.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Graphical models for interactive POMDPs: representations and solutions

Doshi

Zeng

Chen

2008

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

show abstract

“…The Markov property does not hold in these circumstances. In such problems, it can be shown (Lovejoy, 1991;Sondik, 1978;Smallwood and Sondik, 1973;Monahan, 1982;White III and Scherer, 1989;Lin et al, 2004) that it is sufficient to keep a probability distribution called the belief state distribution of the current state. The most challenging aspect of this problem is that the state space becomes continuous, and consequently it is difficult to solve the problem exactly even for a handful of states.…”

Section: Semi-markov Decision Problemsmentioning

confidence: 99%

Reinforcement Learning: A Tutorial Survey and Recent Advances

Gosavi

2009

INFORMS Journal on Computing

252

View full text Add to dashboard Cite

In the last few years, Reinforcement Learning (RL), also called adaptive (or approximate) dynamic programming (ADP), has emerged as a powerful tool for solving complex sequential decision-making problems in control theory. Although seminal research in this area was performed in the artificial intelligence (AI) community, more recently, it has attracted the attention of optimization theorists because of several noteworthy success stories from operations management. It is on large-scale and complex problems of dynamic optimization, in particular the Markov decision problem (MDP) and its variants, that the power of RL becomes more obvious. It has been known for many years that on large-scale MDPs, the curse of dimensionality and the curse of modeling render classical dynamic programming (DP) ineffective. The excitement in RL stems from its direct attack on these curses, allowing it to solve problems that were considered intractable, via classical DP, in the past. The success of RL is due to its strong mathematical roots in the principles of DP, Monte Carlo simulation, function approximation, and AI. Topics treated in some detail in this survey are: Temporal differences, Q-Learning, semi-MDPs and stochastic games. Several recent advances in RL, e.g., policy gradients and hierarchical RL, are covered along with references. Pointers to numerous examples of applications are provided. This overview is aimed at uncovering the mathematical roots of this science, so that readers gain a clear understanding of the core concepts and are able to use them in their own research. The survey points to more than 100 references from the literature.

show abstract

“…This has been done in [9], where efficient algorithms are also proposed for solving the dynamic programming equations.…”

Section: Theorem 1 [1]mentioning

confidence: 99%