Scalable Deep Reinforcement Learning Algorithms for Mean Field Games

Abstract

Mean Field Games (MFGs) have been introduced to efficiently approximate games with very large populations of strategic agents. Recently, the question of learning equilibria in MFGs has gained momentum, particularly using model-free reinforcement learning (RL) methods. One limiting factor to further scale up using RL is that existing algorithms to solve MFGs require the mixing of approximated quantities such as strategies or q-values. This is non-trivial in the case of non-linear function approximation that enjoy good generalization properties, e.g. neural networks. We propose two methods to address this shortcoming. The first one learns a mixed strategy from distillation of historical data into a neural network and is applied to the Fictitious Play algorithm. The second one is an online mixing method based on regularization that does not require memorizing historical data or previous estimates. It is used to extend Online Mirror Descent. We demonstrate numerically that these methods efficiently enable the use of Deep RL algorithms to solve various MFGs. In addition, we show that these methods outperform SotA baselines from the literature. * Equal contribution. † Equal contribution.

Recent Developments in Machine Learning Methods for Stochastic Control and Games

“…A few works have used deep RL methods to compute the best response. For example, DDPG have been used in [84], soft actor-critic (SAC) has been used for a flocking model in [208], while deep Q-learning or some variants of it has been used in [71,207,178]. Recently, several works have studied the advantages and the limitations brought by the regularization of the policy through penalization terms in the cost function [10,71,113].…”

Developing, evaluating and scaling learning agents in multi-agent environments

“…We introduced and studied the convergence to a Nash equilibrium of several scalable learning algorithms for Mean Field Games, together with more specific applications to flocking [24] or vehicles traffic routing management [25]. Our algorithms rely on Fictitious Play [26,27] or Online Mirror Descent algorithms [28] and can be efficiently combined with Deep Reinforcement Learning [29]. Using a well chosen enlarged class of policies, pinned as master policies, allows us to generalise efficiently to several initial population distributions [30].…”

Recent Developments in Machine Learning Methods for Stochastic Control and Games

“…A few works have used deep RL methods to compute the best response. For example, DDPG has been used in [64], soft actor-critic (SAC) has been used for a flocking model in [151], while deep Q-learning or some variants of it has been used in [58,150,130]. Recently, several works have have studied the advantages and the limitations brought by regularization of the policy through penalization terms in the cost function [10,58,91].…”