McKean-Vlasov optimal control: the dynamic programming principle

Djete, Mao Fabrice; Possamaï, Dylan; Tan, Xiaolu

doi:10.48550/arxiv.1907.08860

Cited by 17 publications

(34 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Without the density existence assumption, but under some regularity conditions on the coefficient functions, the DPP has been proved in Pham and Wei [70; 69], and Bayraktar, Cosso, and Pham [9] in different situations. Using abstract measurable selection arguments, a general DPP has been established under minimal conditions in our accompanying article Djete, Possamaï, and Tan [24].…”

Section: Introductionmentioning

confidence: 99%

McKean-Vlasov optimal control: limit theory and equivalence between different formulations

Djete,

Possamaï,

Tan

2020

Preprint

Self Cite

View full text Add to dashboard Cite

We study a McKean-Vlasov optimal control problem with common noise, in order to establish the corresponding limit theory, as well as the equivalence between different formulations, including the strong, weak and relaxed formulation. In contrast to the strong formulation, where the problem is formulated on a fixed probability space equipped with two Brownian filtrations, the weak formulation is obtained by considering a more general probability space with two filtrations satisfying an (H)-hypothesis type condition from the theory of enlargement of filtrations. When the common noise is uncontrolled, our relaxed formulation is obtained by considering a suitable controlled martingale problem. As for classical optimal control problems, we prove that the set of all relaxed controls is the closure of the set of all strong controls, when considered as probability measures on the canonical space. Consequently, we obtain the equivalence of the different formulations of the control problem, under additional mild regularity conditions on the reward functions. This is also a crucial technical step to prove the limit theory of the McKean-Vlasov control problem, that is to say proving that it consists in the limit of a large population control problem with common noise.

show abstract

Section: Introductionmentioning

confidence: 99%

McKean-Vlasov optimal control: limit theory and equivalence between different formulations

Djete,

Possamaï,

Tan

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Since the dynamics is of MKV type, in general, the value function in a MFC problem is the value function of the social planner and it takes the distribution ν as input, see e.g. (Laurière and Pironneau, 2014;Pham and Wei, 2016;Carmona et al, 2019c;Motte and Pham, 2019;Gu et al, 2019;Djete et al, 2019). However, when the population is already evolving according to the sequence of distributions ν α generated by a control α, the cost-to-go of an infinitesimal agent starting at position x and using control α too is simply a function of its position and is given by…”

Section: Discrete Formulationmentioning

confidence: 99%

Reinforcement Learning for Mean Field Games, with Applications to Economics

Angiuli¹,

Fouque²,

Laurière³

2021

Preprint

View full text Add to dashboard Cite

Mean field games (MFG) and mean field control problems (MFC) are frameworks to study Nash equilibria or social optima in games with a continuum of agents. These problems can be used to approximate competitive or cooperative games with a large finite number of agents and have found a broad range of applications, in particular in economics. In recent years, the question of learning in MFG and MFC has garnered interest, both as a way to compute solutions and as a way to model how large populations of learners converge to an equilibrium. Of particular interest is the setting where the agents do not know the model, which leads to the development of reinforcement learning (RL) methods. After reviewing the literature on this topic, we present a two timescale approach with RL for MFG and MFC, which relies on a unified Q-learning algorithm. The main novelty of this method is to simultaneously update an action-value function and a distribution but with different rates, in a model-free fashion. Depending on the ratio of the two learning rates, the algorithm learns either the MFG or the MFC solution. To illustrate this method, we apply it to a mean field problem of accumulated consumption in finite horizon with HARA utility function, and to a trader's optimal liquidation problem.

show abstract

“…s. Despite possibly appearing tangential at first glance, carefully formulated compatibility conditions are essential for dealing with weak solutions for SDEs [41], McKean-Vlasov equations [38,56], control problems [28,29], and mean field games [18,17]. For this paper, we favor the most concise formulation…”

Section: Compatibility Preliminariesmentioning

confidence: 99%

“…The few recent papers on mean field control with common noise, such as [60,28] have proposed various notions of admissible controls, usually with the goal of deriving a dynamic programming principle and an associated Hamilton-Jacobi-Bellman equation. See also [17,Chapter I.6] for the case without common noise.…”

Section: Application To Controlled Mckean-vlasov Dynamicsmentioning

confidence: 99%

Superposition and mimicking theorems for conditional McKean-Vlasov equations

Lacker¹,

Shkolnikov²,

Zhang³

2020

Preprint

View full text Add to dashboard Cite

We consider conditional McKean-Vlasov stochastic differential equations (SDEs), such as the ones arising in the large-system limit of mean field games and particle systems with mean field interactions when common noise is present. The conditional time-marginals of the solutions to these SDEs satisfy non-linear stochastic partial differential equations (SPDEs) of the second order, whereas the laws of the conditional time-marginals follow Fokker-Planck equations on the space of probability measures. We prove two superposition principles: The first establishes that any solution of the SPDE can be lifted to a solution of the conditional McKean-Vlasov SDE, and the second guarantees that any solution of the Fokker-Planck equation on the space of probability measures can be lifted to a solution of the SPDE. We use these results to obtain a mimicking theorem which shows that the conditional time-marginals of an Itô process can be emulated by those of a solution to a conditional McKean-Vlasov SDE with Markovian coefficients. This yields, in particular, a tool for converting open-loop controls into Markovian ones in the context of controlled McKean-Vlasov dynamics.

show abstract

McKean-Vlasov optimal control: the dynamic programming principle

Cited by 17 publications

References 44 publications

McKean-Vlasov optimal control: limit theory and equivalence between different formulations

McKean-Vlasov optimal control: limit theory and equivalence between different formulations

Reinforcement Learning for Mean Field Games, with Applications to Economics

Superposition and mimicking theorems for conditional McKean-Vlasov equations

Contact Info

Product

Resources

About