An Evolutionary Dynamical Analysis of Multi-Agent Learning in Iterated Games

Abstract

In this paper, we investigate Reinforcement learning (RL) in multi-agent systems (MAS) from an evolutionary dynamical perspective. Typical for a MAS is that the environment is not stationary and the Markov property is not valid. This requires agents to be adaptive. RL is a natural approach to model the learning of individual agents. These Learning algorithms are however known to be sensitive to the correct choice of parameter settings for single agent systems. This issue is more prevalent in the MAS case due to the changing interactions amongst the agents. It is largely an open question for a developer of MAS of how to design the individual agents such that, through learning, the agents as a collective arrive at good solutions. We will show that modeling RL in MAS, by taking an evolutionary game theoretic point of view, is a new and potentially successful way to guide learning agents to the most suitable solution for their task at hand. We show how evolutionary dynamics (ED) from Evolutionary Game Theory can help the developer of a MAS in good choices of parameter settings of the used RL algorithms. The ED essentially predict the equilibriums outcomes of the MAS where the agents use individual RL algorithms. More specifically, we show how the ED predict the learning trajectories of Q-Learners for iterated games. Moreover, we apply our results to (an extension of) the COllective INtelligence framework (COIN). COIN is a proved engineering approach for learning of cooperative tasks in MASs. The utilities of the agents are re-engineered to contribute to the global utility. We show how the improved results for MAS RL in COIN, and a developed extension, are predicted by the ED.

Multi-Agent Systems Applications in Energy Optimization Problems: A State-of-the-Art Review

“…Some of the pioneering works in the use of gamification techniques were those proposed by Tuyls et al [77,78] in which they used Reinforcement Learning (RL) to model the learning of the agents. These RL algorithms are sensitive to the correct choice of parameters so it is necessary to make a correct choice of parameter settings.…”

Replicator Dynamics for Multi-agent Learning: An Orthogonal Approach

“…However, it is worth mentioning that other learning algorithms can be described by similar differential equations, e.g. Q-learning with a Boltzmann exploration scheme has been shown to converge to an extension of the replicator dynamics [11]. The evolutionary description of reinforcement learning is detailed in the following subsections using the example of Cross learning.…”

“…They assume this population to evolve such that successful strategies with higher payoffs grow while less successful ones decay. These dynamics are formally connected to reinforcement learning [1,10,11]. Assume a two-player normal form game with payoff matrices A and B for player one and two respectively, and let the policies of player one and two be represented by the probability vectors x = (x 1 , .…”

“…This reduces the theoretical framework available for multi-agent learning. More recently, evolutionary game theory with less strong assumptions than classical game theory could be linked to reinforcement learning and provides useful insights into the learning dynamics [1,3,11].…”

Entertainment Computing in the Orbit

“…Over the last few years the combination of Evolutionary Game Theory (EGT) and Reinforcement Learning (RL) have proved to be powerful theories for designing autonomous agents, and under-standing interactions in systems composed of such agents (Tuyls et al 2006). Modeling agents for an assistive tool requires a thorough understanding of the type and form of interactions within the virtual environment and other agents in the tool controlled by users.…”