Solving high-dimensional Hamilton–Jacobi–Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space

Abstract

Optimal control of diffusion processes is intimately connected to the problem of solving certain Hamilton–Jacobi–Bellman equations. Building on recent machine learning inspired approaches towards high-dimensional PDEs, we investigate the potential of iterative diffusion optimisation techniques, in particular considering applications in importance sampling and rare event simulation, and focusing on problems without diffusion control, with linearly controlled drift and running costs that depend quadratically on the control. More generally, our methods apply to nonlinear parabolic PDEs with a certain shift invariance. The choice of an appropriate loss function being a central element in the algorithmic design, we develop a principled framework based on divergences between path measures, encompassing various existing methods. Motivated by connections to forward-backward SDEs, we propose and study the novel log-variance divergence, showing favourable properties of corresponding Monte Carlo estimators. The promise of the developed approach is exemplified by a range of high-dimensional and metastable numerical examples.

Reachability Analysis of Randomly Perturbed Hamiltonian Systems

“…It turns out, however, that the underlying nonlinear control problem belongs to a class of so-called linearly-solvable stochastic control problems that have been studied by Dvijotham and Todorov (2012), Schütte et al (2012) and others and that can be solved by other means (cf. also Nüsken and Richter (2020)). The corresponding theory goes back to Fleming and co-workers, and we mention only the seminal article Fleming (1977).…”

Deterministic particle flows for constraining stochastic nonlinear systems

“…In this work, we introduced a novel methodological framework for identifying optimal dynamical interventions for constraining diffusive systems. Distinctively from previous work [26,27,44,78] that devises optimal control protocols by employing iterative optimisation procedures, here, we obtained the required interventions in a deterministic and noniterative way. In particular, we showed that splitting the time-resolved constraining information into retrospective and prospective parts, allows for a representation of the optimal interventions in terms of the difference of logarithmic gradients (scores) of two forward probability flows.…”

Interpolating between BSDEs and PINNs: deep learning for elliptic and parabolic boundary value problems

“…We note in passing that Proposition 3.3 extends straightforwardly to L t,v diffusion,int under the assumption that v satisfies appropriate Lipschitz and growth conditions. Similar considerations apply for the BSDE loss, noting that for solutions to the generalized BSDE system [60]…”

“…refers to an appropriate function class, usually consisting of deep neural networks. With a loss function at hand we can apply gradient-descent type algorithms to minimize (estimator versions of) L, keeping in mind that different choices of losses lead to different statistical and computational properties and therefore potentially to different convergence speeds and robustness behaviours [60].…”

“…A variational formulation termed deep BSDE has been first introduced in [19,28], aiming at PDE solutions at a single point, with some variants following e.g. in [60,67]. For an analysis of the approximation error of the deep BSDE method we refer to [29] and for a fully nonlinear version based on second-order BSDEs to [5].…”