α-Rank: Multi-Agent Evaluation by Evolution

Abstract

We introduce α - Rank , a principled evolutionary dynamics methodology, for the evaluation and ranking of agents in large-scale multi-agent interactions, grounded in a novel dynamical game-theoretic solution concept called Markov - Conley chains (MCCs). The approach leverages continuous-time and discrete-time evolutionary dynamical systems applied to empirical games, and scales tractably in the number of agents, in the type of interactions (beyond dyadic), and the type of empirical games (symmetric and asymmetric). Current models are fundamentally limited in one or more of these dimensions, and are not guaranteed to converge to the desired game-theoretic solution concept (typically the Nash equilibrium). α -Rank automatically provides a ranking over the set of agents under evaluation and provides insights into their strengths, weaknesses, and long-term dynamics in terms of basins of attraction and sink components. This is a direct consequence of the correspondence we establish to the dynamical MCC solution concept when the underlying evolutionary model’s ranking-intensity parameter, α , is chosen to be large, which exactly forms the basis of α -Rank. In contrast to the Nash equilibrium, which is a static solution concept based solely on fixed points, MCCs are a dynamical solution concept based on the Markov chain formalism, Conley’s Fundamental Theorem of Dynamical Systems, and the core ingredients of dynamical systems: fixed points, recurrent sets, periodic orbits, and limit cycles. Our α -Rank method runs in polynomial time with respect to the total number of pure strategy profiles, whereas computing a Nash equilibrium for a general-sum game is known to be intractable. We introduce mathematical proofs that not only provide an overarching and unifying perspective of existing continuous- and discrete-time evolutionary evaluation models, but also reveal the formal underpinnings of the α -Rank methodology. We illustrate the method in canonical games and empirically validate it in several domains, including AlphaGo, AlphaZero, MuJoCo Soccer, and Poker.

From Darwin to Poincaré and von Neumann: Recurrence and Cycles in Evolutionary and Algorithmic Game Theory

“…where we used one fundamental property of full-support Nash equilibria, that is here u i,αi (x * ) = u i (x * ), to go from (28) to (29).…”

“…This approach shifts attention from Nash equilibria to a more general notion of recurrence, called chain recurrence, that generalizes both periodicity and Poincaré recurrence. [29] embeds this approach within an algorithmically tractable framework and uses it to develop new training algorithms for multi-agent AI settings.…”

Metagame Autobalancing for Competitive Multiplayer Games

“…An interesting meta-game definition that has recently received attention in multiagent system analysis [4] defines a normal form game over a population of agents π, such that the action set of each player corresponds to choosing an agent π i ∈ π from the population to play the game for them. How these agents were created is not relevant to us; these agents could use hand-crafted heuristics, be trained with reinforcement learning, evolutionary algorithms or any other method.…”

“…Given an evaluation matrix A π computed from a set of strategies (or agents) π, let its response graph [6] represent the dynamics [4] between agents in π. That is, a representation of which strategies (or agents) perform favourably against which other strategies in π.…”

“…The magnitude of rewards (1), (2), and (3) varied between 0-10, 10-99, and 1000 respectively so as to represent a hierarchy of goals for the agent to follow. 4 We have open-sourced our MCTS implementation: https://www.github.com/Danielhp95/Regym VII. RESULTS Using Algorithm 1 defined in Section III, we found the following parameter vectors θ f air and θ cyclic , corresponding to the meta-game balancing defined in Figure 6a and Figure 6b respectively.…”