Generalized multiagent learning with performance bound

Abstract

We present new Multiagent learning (MAL) algorithms with the general philosophy of policy convergence against some classes of opponents but otherwise ensuring high payoffs. We consider a 3-class breakdown of opponent types: (eventually) stationary, self-play and "other" (see Definition 4) agents. We start with ReDVaLeR that can satisfy policy convergence against the first two types and no-regret against the third, but it needs to know the type of the opponents. This serves as a baseline to delineate the difficulty of achieving these goals. We show that a simple modification on ReDVaLeR yields a new algorithm, RV σ (t) , that achieves no-regret payoffs in all games, and convergence to Nash equilibria in self-play (and to best response against eventually stationary opponents -a corollary of no-regret) simultaneously, without knowing the opponent types, but in a smaller class of games than ReDVaLeR . RV σ (t) effectively ensures the performance of a learner during the process of learning, as opposed to the performance of a learned behavior. We show that the expression for regret of RV σ (t) can have a slightly better form than those of other comparable algorithms like GIGA and GIGA-WoLF though, contrastingly, our analysis is in continuous time. Moreover, experiments show that RV σ (t) can converge to an equilibrium in some cases where GIGA, GIGA-WoLF would fail, and to better equilibria where GIGA, GIGA-WoLF converge to undesirable equilibria (coordination games). This important class of coordination games also highlights the key desirability of policy convergence as a criterion for MAL in self-play instead of high average payoffs. To our knowledge, this is also the first successful (guaranteed) attempt at policy convergence of a no-regret algorithm in the Shapley game.

Multiagent meta-level control for radar coordination

“…Several MARL algorithms have been proposed and studied [5,11,18,22], all of which have some theoretical results of convergence in general-sum games. A common assumption of these algorithms is that an agent (or player) knows its own payoff matrix.…”

“…A common assumption of these algorithms is that an agent (or player) knows its own payoff matrix. To guarantee convergence, each algorithm has it own additional assumptions, such as requiring an agent to know a Nash Equilibrium and the strategy of the other players [5,11,18], or observe what actions other agents executed and what rewards they received [18,22]. For practical applications, these assumptions are very constraining and unlikely to hold, and, instead, an agent can only observe the immediate reward after selecting and performing an action.…”

Effective learning in the presence of adaptive counterparts

“…In a recent work, Chakraborty and Sen [21] proposed modeling the learning environments induced by gradient ascent learners (WoLF-IGA, WoLF-PHC [7] and ReDVaLeR [6]) as MDPs. In the presence of a gradient ascent adversary, the learning algorithm, called MB-AIM-FSI, first creates a set of hypotheses about the model of the learning environment that can be induced by the learning adversary.…”

“…Certain approaches also require the observability of the actions made by the other agents [2,3]. Finally, in the less realistic settings, the strategies (i.e., the probability distributions over actions) or the rewards obtained by the other agents are also assumed to be observable [4][5][6].…”

Multi-objects scalable coordinated learning in internet of things

“…The Multi-Agent Reinforcement Learning (MARL) in [5] provides a common approach for solving multi-objects decision-making problems that allows objects to dynamically adapt to changes in the IoT environment. Several MARL algorithms have been proposed in [4,[6][7][8], all of which have some theoretical results of convergence in general-sum games. A common assumption of these algorithms is that an player knows its own payoff matrix.…”