Learning Deep Mean Field Games for Modeling Large Population Behavior

Yang, Jiachen; Ye, Xiaojing; Trivedi, Rakshit; Xu, Huan; Zha, Hongyuan

doi:10.48550/arxiv.1711.03156

Cited by 11 publications

(9 citation statements)

References 14 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[20] shows that under certain conditions, fictitious play converges to a MFE. [28] proposed a deep mean field game to model strategic interactions among the players. [18] proposed a modelfree non stationary algorithm to compute MFE.…”

Section: Related Literaturementioning

confidence: 99%

Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents

Ghosh¹,

Aggarwal²

2020

Preprint

View full text Add to dashboard Cite

We consider a multi-agent Markov strategic interaction over an infinite horizon where agents can be of multiple types. We model the strategic interaction as a mean-field game in the asymptotic limit when the number of agents of each type becomes infinite. Each agent has a private state; the state evolves depending on the distribution of the state of the agents of different types and the action of the agent. Each agent wants to maximize the discounted sum of rewards over the infinite horizon which depends on the state of the agent and the distribution of the state of the leaders and followers. We seek to characterize and compute a stationary multi-type Mean field equilibrium (MMFE) in the above game. We characterize the conditions under which a stationary MMFE exists. Finally, we propose Reinforcement learning (RL) based algorithm using policy gradient approach to find the stationary MMFE when the agents are unaware of the dynamics. We, numerically, evaluate how such kind of interaction can model the cyber attacks among defenders and adversaries, and show how RL based algorithm can converge to an equilibrium.

show abstract

Section: Related Literaturementioning

confidence: 99%

Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents

Ghosh¹,

Aggarwal²

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Model-based approaches to the mean-field approximation require knowledge of model parameters (see, for example, Elliott et al, 2013;Mahajan, 2015, 2016;Li et al, 2017), while model-free methods (Yang et al, 2017 only come with algorithms with asymptotic analysis. In contrast, our method is model-free and comes with provable global non-asymptotic convergence analysis.…”

Section: Related Workmentioning

confidence: 99%

Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator

Luo

Yang

Wang³

et al. 2019

Preprint

View full text Add to dashboard Cite

Multi-agent reinforcement learning has been successfully applied to a number of challenging problems. Despite these empirical successes, theoretical understanding of different algorithms is lacking, primarily due to the curse of dimensionality caused by the exponential growth of the state-action space with the number of agents. We study a fundamental problem of multi-agent linear quadratic regulator in a setting where the agents are partially exchangeable. In this setting, we develop a hierarchical actor-critic algorithm, whose computational complexity is independent of the total number of agents, and prove its global linear convergence to the optimal policy. As linear quadratic regulators are often used to approximate general dynamic systems, this paper provided an important step towards better understanding of general hierarchical mean-field multi-agent reinforcement learning.

show abstract

“…To scale up learning in MARL problem in the presence of a large number, even infinitely many agents, mean-field approximation has been explored to directly model the population behavior of the agents. Yang et al (2017) consider a mean-field game with deterministic linear state transitions, and show it can be reformulated as a mean-field MDP, with mean-field state residing in a finite-dimensional probability simplex. Yang et al (2018) take the mean-field approximation over actions, such that the interaction for any given agent and the population is approximated by the interaction between the agent's action and the averaged actions of its neighboring agents.…”

Section: Introductionmentioning

confidence: 99%

Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach

Li¹,

Wang²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Multi-agent reinforcement learning (MARL) becomes more challenging in the presence of more agents, as the capacity of the joint state and action spaces grows exponentially in the number of agents. To address such a challenge of scale, we identify a class of cooperative MARL problems with permutation invariance, and formulate it as mean-field Markov decision processes (MDP). To exploit the permutation invariance therein, we propose the meanfield proximal policy optimization (MF-PPO) algorithm, at the core of which is a permutationinvariant actor-critic neural architecture. We prove that MF-PPO attains the globally optimal policy at a sublinear rate of convergence. Moreover, its sample complexity is independent of the number of agents. We validate the theoretical advantages of MF-PPO with numerical experiments in the multi-agent particle environment (MPE). In particular, we show that the inductive bias introduced by the permutation-invariant neural architecture enables MF-PPO to outperform existing competitors with a smaller number of model parameters, which is the key to its generalization performance.

show abstract

Learning Deep Mean Field Games for Modeling Large Population Behavior

Cited by 11 publications

References 14 publications

Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents

Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents

Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator

Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach

Contact Info

Product

Resources

About