Backward Deep BSDE Methods and Applications to Nonlinear Problems

Yu, Yajie; Hientzsch, Bernhard; Ganesan, Narayan

doi:10.48550/arxiv.2006.07635

Cited by 1 publication

(3 citation statements)

References 8 publications

(9 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…( 26) is equivalent to minimize the measurability loss defined by Eq. (19). Therefore, Theorem 1 justifies that the Deep BSDE method unexpectedly fits the value function gradients.…”

Section: Related To Recent Advancesmentioning

confidence: 70%

“…It is interesting that if Eq. ( 19) holds at t = 0, then it is sufficient enough to claim that y θ t is adapted, and consequently (19) holds on the whole interval [0, T ]. One can easily verify this from the following observation.…”

Section: Deep Bsde-ml Methodsmentioning

confidence: 99%

“…However, our theoretical result suggests that this parameterizing strategy fits the value function gradients, not the value function itself. [19] has mentioned this variance form loss Eq. ( 22) and its variants, primarily based on considerations of numerical implementations.…”

Section: Related To Recent Advancesmentioning

confidence: 99%

See 2 more Smart Citations

Deep BSDE-ML Learning and Its Application to Model-Free Optimal Control

Wang¹,

Ni²

2022

Preprint

View full text Add to dashboard Cite

A modified Deep BSDE (backward differential equation) learning method with measurability loss, called Deep BSDE-ML method, is introduced in this paper to solve a kind of linear decoupled forward-backward stochastic differential equations (FBSDEs), which is encountered in the policy evaluation of learning the optimal feedback policies of a class of stochastic control problems. The measurability loss is characterized via the measurability of BSDE's state at the forward initial time, which differs from that related to terminal state of the known Deep BSDE method. Though the minima of the two loss functions are shown to be equal, this measurability loss is proved to be equal to the expected mean squared error between the true diffusion term of BSDE and its approximation. This crucial observation extends the application of the Deep BSDE methodapproximating the gradients of the solution of a partial differential equation (PDE) instead of the solution itself.Simultaneously, a learning-based framework is introduced to search an optimal feedback control of a deterministic nonlinear system. Specifically, by introducing Gaussian exploration noise, we are aiming to learn a robust optimal controller under this stochastic case. This reformulation sacrifices the optimality to some extent, but as suggested in reinforcement learning (RL) exploration noise is essential to enable the model-free learning. The new stochastic optimal control problem is solved with general policy iteration methodology-repeating policy evaluation and policy improvment. Instead of fitting the value function, our policy evaluation approximates its gradient, thus can be seamlessly integrated with policy improvement without manually differentiating a neural network. By using the proposed Deep BSDE-ML method, this is achieved through optimizing the understood loss function in the FBSDE formulation. With some simulating tricks, the whole algorithm can be implemented in both model-based and model-free fashions. Compared with the Markov framework of RL, our method is built on the diffusion process, thus is preferred from a theoretical point of view for continuous-time and continuous-space tasks. Numerical Experiments suggest that the proposed model-free approach performs as good as its model-based counterpart.

show abstract

“…( 26) is equivalent to minimize the measurability loss defined by Eq. (19). Therefore, Theorem 1 justifies that the Deep BSDE method unexpectedly fits the value function gradients.…”

Section: Related To Recent Advancesmentioning

confidence: 70%

Section: Deep Bsde-ml Methodsmentioning

confidence: 99%