Solving Transition Independent Decentralized Markov Decision Processes

Abstract

Formal treatment of collaborative multi-agent systems has been lagging behind the rapid progress in sequential decision making by individual agents. Recent work in the area of decentralized Markov Decision Processes (MDPs) has contributed to closing this gap, but the computational complexity of these models remains a serious obstacle. To overcome this complexity barrier, we identify a specific class of decentralized MDPs in which the agents' transitions are independent. The class consists of independent collaborating agents that are tied together through a structured global reward function that depends on all of their histories of states and actions. We present a novel algorithm for solving this class of problems and examine its properties, both as an optimal algorithm and as an anytime algorithm. To our best knowledge, this is the first algorithm to optimally solve a non-trivial subclass of decentralized MDPs. It lays the foundation for further work in this area on both exact and approximate algorithms.

Optimizing fixed-size stochastic controllers for POMDPs and decentralized POMDPs

“…Some examples include distributed robot control [2,9] and networking problems [3,12]. In multiagent domains, robot sensors not only provide uncertain and incomplete information about their own state, but also about the location of the other robots.…”

Planning in stochastic domains for multiple agents with individual continuous resource state-spaces

“…[18] proposes a solution based on Dynamic Programming (DP), while [19] extends point-based DP to the case of decentralized agents, and [20] applies heuristic search. Work in [21] has focused on a simpler model, denoted transition-independent where agents do not affect each other's state but cooperate via a joint reward signal instead. The difference between these algorithms and our work lies in their assumption that the agents' internal states are discrete and their modeling of a low number of action outcomes.…”

“…The assumption is realistic since many proprioceptive signals are accessible to the measure of the agent itself but are not communicated to other agents. Therefore the approach in [21] is of particular interest to us in this paper. Our model of interest is thus that of transition independent DECHMDPs (TI-DEC-HMDPs), where agents do not communicate but are subjected to internal resource constraints.…”

“…Our model of interest is thus that of transition independent DECHMDPs (TI-DEC-HMDPs), where agents do not communicate but are subjected to internal resource constraints. It follows that our theoretical approach remains within the brackets of [21]. This approach is attractive because it allows to consider each agent individually: the core idea is to compute the coverage set of each agent, that is the set that regroups the agent strategies such that all of the possible behaviors of the other agents are optimally covered.…”

Using evolution strategies to solve DEC-POMDP problems

“…Recently, several researchers have concentrated on finding approximate solutions or finding new subsets of DEC-POMDP problems which are easier to solve and can model some real world problems (Becker et al 2004;Goldman and Zilberstein 2004). Although current approximate solution algorithms are able to solve slightly larger problems than exact solution algorithms, they still need considerable improvement in order to handle real world problems involving very large state spaces (Bernstein et al 2005;Nair et al 2003).…”