Learning with Opponent-Learning Awareness

Abstract

Multi-agent settings are quickly gathering importance in machine learning. This includes a plethora of recent work on deep multiagent reinforcement learning, but also can be extended to hierarchical reinforcement learning, generative adversarial networks and decentralised optimization. In all these settings the presence of multiple learning agents renders the training problem non-stationary and often leads to unstable training or undesired final results. We present Learning with Opponent-Learning Awareness (LOLA), a method in which each agent shapes the anticipated learning of the other agents in the environment. The LOLA learning rule includes an additional term that accounts for the impact of one agent's policy on the anticipated parameter update of the other agents. Preliminary results show that the encounter of two LOLA agents leads to the emergence of tit-for-tat and therefore cooperation in the iterated prisoners' dilemma (IPD), while independent learning does not. In this domain, LOLA also receives higher payouts compared to a naive learner, and is robust against exploitation by higher order gradient-based methods. Applied to infinitely repeated matching pennies, LOLA agents converge to the Nash equilibrium. In a round robin tournament we show that LOLA agents can successfully shape the learning of a range of multi-agent learning algorithms from literature, resulting in the highest average returns on the IPD. We also show that the LOLA update rule can be efficiently calculated using an extension of the likelihood ratio policy gradient estimator, making the method suitable for model-free reinforcement learning. This method thus scales to large parameter and input spaces and nonlinear function approximators. We also apply LOLA to a grid world task with an embedded social dilemma using deep recurrent policies and opponent modelling. Again, by explicitly considering the learning of the other agent, LOLA agents learn to cooperate out of self-interest. Our code is available at github.com/alshedivat/lola.

Social physics

“…Ref. [625] shows that naive and commonly defecting reinforcement learners start to cooperate when they incorporate in their own learning process the awareness of their opponent's learning. Appropriately dubbed learning with opponent-learning awareness or LOLA, the approach leads to the emergence of tit-for-tat and consequent cooperation in the iterated prisoners' dilemma.…”

“…One of the best-performing strategies in terms of the overall average score is the Desired Belief Strategy [632], which actively analyses the opponent and responds depending on whether the opponent's action is perceived as noise or a genuine behavioural change. Ultimately, an inescapable conclusions is that reinforcement learning is an effective means to construct strong strategies for various iterated social-dilemma situations [625,631,633,634].…”

Provably Efficient Policy Optimization for Two-Player Zero-Sum Markov Games

“…For example, Lockhart et al [2019] performs direct policy optimization against worst-case opponents and effectively finds an NE in Kuhn Poker and Goofspiel card game. Foerster et al [2017] invented LOLA where each agent shapes learning of other agents. It gave the highest average returns on the iterated prisoners' dilemma (IPD).…”

Cooperative Autonomous Vehicles that Sympathize with Human Drivers

“…In other words, we perform our experiments in scenarios with a similar nature to the one depicted in Figure 1 that essentially require all agents to work together and success cannot be achieved by any of them individually. Our main contributions are as follows: [8]. Additionally, the idea of decorrelating training samples by drawing them from an experience replay buffer becomes obsolete and a multi-agent derivation of importance sampling can be employed to remove the outdated samples from the replay buffer [9].…”