Reinforcement Learning for Mean Field Games, with Applications to Economics

Abstract

Mean field games (MFG) and mean field control problems (MFC) are frameworks to study Nash equilibria or social optima in games with a continuum of agents. These problems can be used to approximate competitive or cooperative games with a large finite number of agents and have found a broad range of applications, in particular in economics. In recent years, the question of learning in MFG and MFC has garnered interest, both as a way to compute solutions and as a way to model how large populations of learners converge to an equilibrium. Of particular interest is the setting where the agents do not know the model, which leads to the development of reinforcement learning (RL) methods. After reviewing the literature on this topic, we present a two timescale approach with RL for MFG and MFC, which relies on a unified Q-learning algorithm. The main novelty of this method is to simultaneously update an action-value function and a distribution but with different rates, in a model-free fashion. Depending on the ratio of the two learning rates, the algorithm learns either the MFG or the MFC solution. To illustrate this method, we apply it to a mean field problem of accumulated consumption in finite horizon with HARA utility function, and to a trader's optimal liquidation problem.

Deep Learning for Mean Field Games and Mean Field Control with Applications to Finance

“…Let us stress that this problem is an extended MFC: the population distribution is not frozen during the optimization over α, and the interactions occur through the distribution of controls ν α . This model is solved by reinforcement learning techniques (for both MFC and the corresponding MFG) in [9]. Here, we present results obtained with the deep learning method described above, see Algorithm 1.…”

“…While the present chapter concentrates on ML applications of MFGs and MFC in finance, the reader should not be surprised if they recognize a strong commonality of ideas and threads with the subchapter [9] dealing with reinforcement learning for MFGs with a special focus on a two-timescale procedure, and the subchapter [36] offering a review of neural-network-based algorithms for stochastic control and PDE applications in finance.…”

Markov decision processes under model uncertainty

“…The Markov decision problem leads to an infinite horizon stochastic optimal control problem in discrete-time, which finds many applications in finance and economics, compare, for example, Bäuerle and Rieder (2011), Hambly et al (2021), or White (1993) for an overview. It can, among a multitude of other applications, be used to learn the optimal structure of portfolios and the optimal trading behavior, see, for example, Bertoluzzo and Corazza (2012), Chang and Lee (2017), Gold (2003), Hu and Lin (2019), Xiong et al (2018), to learn optimal hedging strategies, see, for example, Angiuli et al (2022), Angiuli et al (2021), Cao et al (2021), Dixon et al (2020), , Halperin (2020), Li et al (2009), Schäl (2002), to optimize inventory-production systems (Uğurlu, 2017), or to study socio-economic systems under the influence of climate change as in Shuvo et al (2020).…”

Markov Decision Processes under Model Uncertainty

“…[9], [13], [19], [24], [40], to learn optimal hedging strategies, see, e.g. [3], [4], [12], [16], [17], [21], [29], [34], or even to study socio-economic systems under the influence of climate change as in [35].…”

“…Next, we introduce a parametric approach in which we assume that the asset returns follow a multivariate normal distribution with unknown parameters. 4 To this end, we build on the setting exposed in Section 4, where m > 1, and where we choose T = R D , and p = 1. Moreover, we consider the following unbiased estimators of mean and covariance…”