Constrained Multiagent Markov Decision Processes: a Taxonomy of Problems and Algorithms

Abstract

In domains such as electric vehicle charging, smart distribution grids and autonomous warehouses, multiple agents share the same resources. When planning the use of these resources, agents need to deal with the uncertainty in these domains. Although several models and algorithms for such constrained multiagent planning problems under uncertainty have been proposed in the literature, it remains unclear when which algorithm can be applied. In this survey we conceptualize these domains and establish a generic problem class based on Markov decision processes. We identify and compare the conditions under which algorithms from the planning literature for problems in this class can be applied: whether constraints are soft or hard, whether agents are continuously connected, whether the domain is fully observable, whether a constraint is momentarily (instantaneous) or on a budget, and whether the constraint is on a single resource or on multiple. Further we discuss the advantages and disadvantages of these algorithms. We conclude by identifying open problems that are directly related to the conceptualized domains, as well as in adjacent research areas.

Deep reinforcement learning for active hypothesis testing with heterogeneous agents and cost constraints

“…This feature is absent when some agents exit earlier than others because the input size of the network changes. We finally wish to point out that, to the best of our knowledge, only tree search algorithms have been proposed for multi-agent POMDPs with constraints [57].…”

Multi-Agent Chance-Constrained Stochastic Shortest Path with Application to Risk-Aware Intelligent Intersection

“…According to the definition above, if X = N , i.e., singleagent interaction points, then agents are independent, except for sharing the risk budget. Such variant has been studied extensively in the literature for constrained MDPs (see [39] for a comprehensive survey). As reported below, we provide an exact computation method, which improves upon the approximate approach provided in [27].…”

Decision-making under uncertainty for multi-robot systems

“…Specifically, we are investigating approaches that ensure the fair allocation of a multi-unit resource to a set of robots. This problem has been considered in the cooperative setting as resource-constrained MAS [8], and we are extending into the non-cooperative setting by developing auction-based approaches with a price structure that ensures that robot bids are truthful. This requires a nonstraightforward adaptation of traditional pricing structures because bids also include the probability of a certain resource use.…”