Evolutionary Dynamics of Multi-Agent Learning: A Survey

Abstract

The interaction of multiple autonomous agents gives rise to highly dynamic and nondeterministic environments, contributing to the complexity in applications such as automated financial markets, smart grids, or robotics. Due to the sheer number of situations that may arise, it is not possible to foresee and program the optimal behaviour for all agents beforehand. Consequently, it becomes essential for the success of the system that the agents can learn their optimal behaviour and adapt to new situations or circumstances. The past two decades have seen the emergence of reinforcement learning, both in single and multi-agent settings, as a strong, robust and adaptive learning paradigm. Progress has been substantial, and a wide range of algorithms are now available. An important challenge in the domain of multi-agent learning is to gain qualitative insights into the resulting system dynamics. In the past decade, tools and methods from evolutionary game theory have been successfully employed to study multi-agent learning dynamics formally in strategic interactions. This article surveys the dynamical models that have been derived for various multi-agent reinforcement learning algorithms, making it possible to study and compare them qualitatively. Furthermore, new learning algorithms that have been introduced using these evolutionary game theoretic tools are reviewed. The evolutionary models can be used to study complex strategic interactions. Examples of such analysis are given for the domains of automated trading in stock markets and collision avoidance in multi-robot systems. The paper provides a roadmap on the progress that has been achieved in analysing the evolutionary dynamics of multi-agent learning by highlighting the main results and accomplishments.

α-Rank: Multi-Agent Evaluation by Evolution

“…This system of coupled differential equations models the temporal dynamics of the populations' strategy profiles when they interact, and can be extended readily to the general K-wise interaction case (see Supplementary Material Section 5.2.2 for more details). The replicator dynamics provide useful insights into the micro-dynamical characteristics of games, revealing strategy flows, basins of attraction, and equilibria [34] when visualized on a trajectory plot over the strategy simplex (e.g, Fig. 4).…”

Evolution of commitment and level of participation in public goods games

“…For large β, imitation becomes increasingly deterministic. It is noteworthy, especially for those who are familiar with other learning literature, that this parameter plays a similar role as the temperature factor in Boltzmann exploration mechanism usually used in Reinforcement Learning to balance between exploitation and exploration [5]. Indeed, as exploration is introduced below, β balances between greedily mimicing more successful interaction partners and randomly switching to the alternatives available in the population.…”

“…5 When facing FAKE players who commit but then do not contribute, COMP F can choose to take immediately the compensation as stated in the agreement thereby ceasing the group interaction for the rest of the commitment time (R − 1). Yet, the commitment player may see that although the expected number F was not attained, there is still sufficient participation to make it worthwhile to continue for the remaining rounds.…”

Prediction-Based Multi-Agent Reinforcement Learning in Inherently Non-Stationary Environments

“…ODHC mitigates dynamic changes in environments (Zeng et al, 2007), and is a hill climbingbased algorithm that explores new peaks where convergence of the search for a better solution is hastened. Evolutionary game theory was employed to gain insight into the environment dynamics in MARL systems (Bloembergen et al, 2015). (Trojanowski and Michalewicz, 1999) memory usage (Cobb, 1990) PS macromutation operator (Esquivel and Coello, 2004) local change discovery (Cui et al, 2009) ACO enhanced communication (Dréo and Siarry, 2006) memory usage (Mavrovouniotis and Yang, 2011) AIS dynamic clonal selection (Kim and Bentley, 2002) Other self-organization and reproduction (Annunziato et al, 2001) Bayesian optimization (Kobliha et al, 2006) …”