Non-Chaotic Limit Sets in Multi-Agent Learning

Abstract

Non-convergence is an inherent aspect of adaptive multi-agent systems, and even basic learning models, such as the replicator dynamics, are not guaranteed to equilibriate. Limit cycles, and even more complicated chaotic sets are in fact possible even in rather simple games, including variants of the Rock-Paper-Scissors game. A key challenge of multi-agent learning theory lies in characterization of these limit sets, based on qualitative features of the underlying game. Although chaotic behavior in learning dynamics can be precluded by the celebrated Poincar\'e-Bendixson theorem, it is only applicable directly to low-dimensional settings. In this work, we attempt to find other characteristics of a game that can force regularity in the limit sets of learning. We show that behavior consistent with the Poincaré-Bendixson theorem (limit cycles, but no chaotic attractor) follows purely from the topological structure of the interaction graph, even for high-dimensional settings with an arbitrary number of players, and arbitrary payoff matrices. We prove our result for a wide class of follow-the-regularized leader (FoReL) dynamics, which generalize replicator dynamics, for binary games characterized interaction graphs where the payoffs of each player are only affected by one other player (i.e., interaction graphs of indegree one). Moreover, we provide simple conditions under which such behavior translates into efficiency guarantees, implying that FoReL learning achievestime-averaged sum of payoffs at least as good as that of a Nash equilibrium, thereby connecting the topology of the dynamics to social-welfare analysis.

Heterogeneous Beliefs and Multi-Population Learning in Network Games

“…On the other hand, networked multi-agent learning constitutes one of the current frontiers in AI and ML research [43,30,16]. Recent theoretical advances on network games provide conditions for learning behaviors to be not chaotic [6,34], and investigate the convergence of Q-learning and continuous-time FP in the case of network competitions [7,28]. However, [7,28] consider that there is only one agent on each vertex, and hence their models are essentially for homogeneous systems.…”

Asymptotic Convergence and Performance of Multi-Agent Q-Learning Dynamics

“…In fact, recent work has consistently found that, when learning on games, agents may present a wide array of behaviours. This includes cycles [1,2,3], and even chaos [4,5,6,7,8]. Furthermore, the equilibria of a game need not be unique, so even if convergence is guaranteed, it may be to one of many (or even a continuum) of equilibria.…”