Reinforcement Learning Algorithm for Mixed Mean Field Control Games

Abstract

We present a new combined Mean Field Control Game (MFCG) problem which can be interpreted as a competitive game between collaborating groups and its solution as a Nash equilibrium between the groups. Within each group the players coordinate their strategies. An example of such a situation is a modification of the classical trader's problem. Groups of traders maximize their wealth. They are faced with transaction cost for their own trades and a cost for their own terminal position. In addition they face a cost for the average holding within their group. The asset price is impacted by the trades of all agents. We propose a reinforcement learning algorithm to approximate the solution of such mixed Mean Field Control Game problems. We test the algorithm on benchmark linear-quadratic specifications for which we have analytic solutions.

Unified reinforcement Q-learning for mean field game and control problems

“…As mentioned above, this paper proves the convergence of the algorithms proposed in [Angiuli et al, 2022b, Angiuli et al, 2022a. The algorithms have been extended to the finite time horizon setting in [Angiuli et al, 2023b] and to deep actor-critic methods in [Angiuli et al, 2023a].…”

“…We then provide an numerical example illustrating the convergence. Last, we generalize our convergence result to a three-timescale RL algorithm introduced in [Angiuli et al, 2022a] to solve mixed Mean Field Control Games (MFCGs).…”

“…In this paper, we analyze multi-timescales algorithms that have been introduced and studied numerically in [Angiuli et al, 2022b, Angiuli et al, 2022a for mean field games, mean field control problems and mixed mean field control games. As in these references, the problems are studied in the context of finite state and action spaces, and in infinite time horizon.…”

Markov decision processes under model uncertainty

“…The Markov decision problem leads to an infinite horizon stochastic optimal control problem in discrete-time, which finds many applications in finance and economics, compare, for example, Bäuerle and Rieder (2011), Hambly et al (2021), or White (1993) for an overview. It can, among a multitude of other applications, be used to learn the optimal structure of portfolios and the optimal trading behavior, see, for example, Bertoluzzo and Corazza (2012), Chang and Lee (2017), Gold (2003), Hu and Lin (2019), Xiong et al (2018), to learn optimal hedging strategies, see, for example, Angiuli et al (2022), Angiuli et al (2021), Cao et al (2021), Dixon et al (2020), , Halperin (2020), Li et al (2009), Schäl (2002), to optimize inventory-production systems (Uğurlu, 2017), or to study socio-economic systems under the influence of climate change as in Shuvo et al (2020).…”

Markov Decision Processes under Model Uncertainty

“…[9], [13], [19], [24], [40], to learn optimal hedging strategies, see, e.g. [3], [4], [12], [16], [17], [21], [29], [34], or even to study socio-economic systems under the influence of climate change as in [35].…”