A fast iterative PDE-based algorithm for feedback controls of nonsmooth mean-field control problems

Reisinger, Christoph; Stockinger, Wolfgang; Zhang, Yufei

doi:10.48550/arxiv.2108.06740

Cited by 6 publications

(10 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this case, (1.7) can be viewed as an infinite-dimensional extension of the iterative shrinkage-thresholding algorithm (see [5,32]).…”

Section: Standing Assumptions and Main Resultsmentioning

confidence: 99%

“…) is the Fréchet derivative of the differentiable component of J(•; ξ 0 ) at the iterate α m (see [39,1]), while the function prox τ ℓ can be identified as the proximal map of the nonsmooth component of J(•; ξ 0 ). We refer the reader to [32] for a detailed derivation of the algorithm and to [21,34] for similar gradient-based algorithms without the nonsmooth term ℓ.…”

Section: Introductionmentioning

confidence: 99%

“…The PPGM (1.7) improves the efficiency of the policy iteration (see [20,18]) and the Method of Successive Approximation (see [23,21]) by avoiding a pointwise minimization of the Hamiltonian over the action space, which may be expensive, especially in a high-dimensional setting. It has been successfully applied to high-dimensional control problems in [32] by solving the linear BSDE (1.9) numerically; see e.g., [16] and [32] and references therein for various numerical schemes.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Linear convergence of a policy gradient method for finite horizon continuous time stochastic control problems

Reisinger¹,

Stockinger²,

Zhang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Despite its popularity in the reinforcement learning community, a provably convergent policy gradient method for general continuous space-time stochastic control problems has been elusive. This paper closes the gap by proposing a proximal gradient algorithm for feedback controls of finite-time horizon stochastic control problems. The state dynamics are continuous time nonlinear diffusions with controlled drift and possibly degenerate noise, and the objectives are nonconvex in the state and nonsmooth in the control. We prove under suitable conditions that the algorithm converges linearly to a stationary point of the control problem, and is stable with respect to policy updates by approximate gradient steps. The convergence result justifies the recent reinforcement learning heuristics that adding entropy regularization to the optimization objective accelerates the convergence of policy gradient methods. The proof exploits careful regularity estimates of backward stochastic differential equations.

show abstract

“…In this case, (1.7) can be viewed as an infinite-dimensional extension of the iterative shrinkage-thresholding algorithm (see [5,32]).…”

Section: Standing Assumptions and Main Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Linear convergence of a policy gradient method for finite horizon continuous time stochastic control problems

Reisinger¹,

Stockinger²,

Zhang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Among the numerical methods proposed for mean-field control problems, we refer to [21] for a policy gradient-type method where feedback controls are approximated by neural networks and optimised for a given objective function; to [32] and again to [21] for a mean-field FBSDE method, generalising the deep BSDE method to mean-field dependence and in the former case to delayed effects; to [48] for a hybrid model where the mean-field distribution is approximated by a particle system and the control is obtained by numerical approximation of a PDE; and to [6] for a survey of methods for the coupled PDE systems, mainly in the spirit of the seminal works [3,2,5]; see also a related semi-Lagrangian scheme in [16]; a gradient method and penalisation approach in [44]; and a recent analysis of policy iteration in [38].…”

Section: Numerics For Mean-field Control Problemsmentioning

confidence: 99%

“…with η ∈ R and δ ≥ 0 such that δ 2 ≤ η 2 . The effect of this modification is that in (48) Λt is replaced by (1 − δ) Λt and additionally the term…”

Section: Solution Via Riccati Odesmentioning

confidence: 99%

Optimal bailout strategies resulting from the drift controlled supercooled Stefan problem

Cuchiero¹,

Reisinger²,

Rigger³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the problem faced by a central bank which bails out distressed financial institutions that pose systemic risk to the banking sector. In a structural default model with mutual obligations, the central agent seeks to inject a minimum amount of cash in order to limit defaults to a given proportion of entities. We prove that the value of the central agent's control problem converges as the number of defaultable institutions goes to infinity, and that it satisfies a drift controlled version of the supercooled Stefan problem. We compute optimal strategies in feedback form by solving numerically a forward-backward coupled system of PDEs. Our simulations show that the central agent's optimal strategy is to subsidise banks whose asset values lie in a non-trivial time-dependent region. Finally, we study a linear-quadratic version of the model where instead of the terminal losses, the agent optimises a terminal cost function of the equity values. In this case, we are able to give semi-analytic strategies, which we again illustrate numerically.

show abstract

Scalable Learning for Spatiotemporal Mean Field Games Using Physics-Informed Neural Operator

Liu,

Chen,

2024

Mathematics

View full text Add to dashboard Cite

This paper proposes a scalable learning framework to solve a system of coupled forward–backward partial differential equations (PDEs) arising from mean field games (MFGs). The MFG system incorporates a forward PDE to model the propagation of population dynamics and a backward PDE for a representative agent’s optimal control. Existing work mainly focus on solving the mean field game equilibrium (MFE) of the MFG system when given fixed boundary conditions, including the initial population state and terminal cost. To obtain MFE efficiently, particularly when the initial population density and terminal cost vary, we utilize a physics-informed neural operator (PINO) to tackle the forward–backward PDEs. A learning algorithm is devised and its performance is evaluated on one application domain, which is the autonomous driving velocity control. Numerical experiments show that our method can obtain the MFE accurately when given different initial distributions of vehicles. The PINO exhibits both memory efficiency and generalization capabilities compared to physics-informed neural networks (PINNs).

show abstract

A fast iterative PDE-based algorithm for feedback controls of nonsmooth mean-field control problems

Cited by 6 publications

References 48 publications

Linear convergence of a policy gradient method for finite horizon continuous time stochastic control problems

Linear convergence of a policy gradient method for finite horizon continuous time stochastic control problems

Optimal bailout strategies resulting from the drift controlled supercooled Stefan problem

Scalable Learning for Spatiotemporal Mean Field Games Using Physics-Informed Neural Operator

Contact Info

Product

Resources

About