Beyond Strict Competition: Approximate Convergence of Multi-agent Q-Learning Dynamics

Abstract

The behaviour of multi-agent learning in competitive settings is often considered under the restrictive assumption of a zero-sum game. Only under this strict requirement is the behaviour of learning well understood; beyond this, learning dynamics can often display non-convergent behaviours which prevent fixed-point analysis. Nonetheless, many relevant competitive games do not satisfy the zero-sum assumption. Motivated by this, we study a smooth variant of Q-Learning, a popular reinforcement learning dynamics which balances the agents' tendency to maximise their payoffs with their propensity to explore the state space. We examine this dynamic in games which are `close' to network zero-sum games and find that Q-Learning converges to a neighbourhood around a unique equilibrium. The size of the neighbourhood is determined by the `distance' to the zero-sum game, as well as the exploration rates of the agents. We complement these results by providing a method whereby, given an arbitrary network game, the `nearest' network zero-sum game can be found efficiently. Importantly, our theoretical guarantees are widely applicable in different game settings, regardless of whether the dynamics ultimately reach an equilibrium, or remain non convergent.

Stability of Multi-Agent Learning in Competitive Networks: Delaying the Onset of Chaos

“…so that each agent has N 0 = N − 1 neighbours. This corresponds also to the case analysed by (Sanders, Farmer, and Galla 2018) and (Hussain, Belardinelli, and Piliouras 2023) in which it was predicted that the boundary between stable and unstable learning dynamics is impacted by the total number of agents.…”

“…In particular, the region in which learning converges to fixed point seems to vanish as the number of agents increases. This result is supported by that of (Hussain, Belardinelli, and Piliouras 2023) in which a lower bound on exploration rates was determined so that Q-Learning dynamics converge to a unique equilibrium. Again it was shown that this lower bound increases with the number of agents.…”

“…In their work, it was found that complex dynamics (cycles and chaos) becomes more prominent as the number of players in the game increases. This is similarly true of (Hussain, Belardinelli, and Piliouras 2023) in which (QLD) was analysed in arbitrary games, without imposing any structure on interactions between agents. The authors similarly concluded that, as the number of agents increases, higher exploration rates T are required to ensure convergence to a QRE.…”

“…In fact, recent work has shown that chaotic dynamics occur in games even slightly perturbed from the zero-sum setting (Galla and Farmer 2013;Sato, Akiyama, and Farmer 2002). In addition, recent work (Hussain, Belardinelli, and Piliouras 2023;Sanders, Farmer, and Galla 2018) has shown that the ability of learning dynamics to reach an equilibrium with low exploration rates diminishes as the number of agents increases. These technical challenges present a strong barrier towards ensuring the convergence of learning in competitive games with many players.…”

“…However, in both (Hussain, Belardinelli, and Piliouras 2023) and (Sanders, Farmer, and Galla 2018) it was assumed that all agents are directly influenced by all other agents in the environment. In practice, however, this does not hold.…”