Replicator Dynamics for Multi-agent Learning: An Orthogonal Approach

Abstract

Today's society is largely connected and many real life applications lend themselves to be modeled as multi-agent systems. Although such systems as well as their models are desirable, e.g. for reasons of stability or parallelism, they are highly complex and therefore difficult to understand or predict. Multi-agent learning has been acknowledged to be indispensable to control or find solutions for such systems. Recently, evolutionary game theory has been linked to multi-agent reinforcement learning. However, gaining insight into the dynamics of games, especially if time dependent, remains a challenging problem. This article introduces a new perspective on the reinforcement learning process described by the replicator dynamics, providing a tool to design time dependent parameters of the game or the learning process. This perspective is orthogonal to the common view of policy trajectories driven by the replicator dynamics. Rather than letting the time dimension collapse, the set of initial policies is considered to be a particle cloud that approximates a distribution and we look at the evolution of this distribution over time. First, the methodology is described, then it is applied to an example game and viable extensions are discussed.

Learning to Resolve Social Dilemmas: A Survey

“…On a related note, and Tuyls et al (2006) considered certain classes of two-player matrix games and derived a connection between the exploration-exploitation scheme of RL and the selection-mutation mechanisms of evolutionary game theory; exploration being analogous to mutation and exploitation to selection. Further, there are links between RL and replicator dynamics; Kaisers and Tuyls (2010) showed empirical confirmation of the match between the learning trajectories of their frequency-adjusted Q-learning (FAQ-learning) algorithm and replicator dynamics for three 2 × 2 games, viz., Prisoner's Dilemma, Battle of Sexes, and Matching Pennies.…”

“…Despite these differences, existing research has found some connections between the three approaches. For example, the learning trajectories for RL and replicator dynamics were shown to match for certain PD games (Kaisers & Tuyls, 2010). However, many of the findings that connect the different learning approaches are not readily relevant to dilemma games.…”

Multi-Agent Deep Reinforcement Learning for Multi-Robot Applications: A Survey

“…The biggest advantage is that each agent/robot can implement its own RL algorithm and there is no need for coordination and, consequently, a joint policy calculation [16,27]. Independent classic Q-learners have shown promising results in AI [29,30], as well as in robotics [31,32]. On the other hand, the joint learners aim to learn the joint optimal policy from O and A.…”

Overlapping Coalition Formation via Probabilistic Topic Modeling

“…The more recent work of [2] explores a PAC (probably approximately correct) model for obtaining theoretical predictions for the value of coalitions that have not been observed in the past. The links between evolutionary game theory and multiagent reinforcement learning is the topic of study in [31,19].…”