Deep reinforcement learning for partial differential equation control

Farahmand, Amir-massoud; Nabi, Saleh; Nikovski, Daniel

doi:10.23919/acc.2017.7963427

Cited by 43 publications

(35 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These issues can be partially addressed by building a hybrid architecture of machine learning and PDE, in which machine learning helps predict or resolve such unknowns. For example, PDE control can be formulated as a reinforcement learning problem ( Farahmand et al., 2017 ). The most extreme condition is that the form of coefficient/equation is unknown, but it can still be learned by machine learning purely from harnessing the experimental and simulation data.…”

Section: Knowledge and Its Representationsmentioning

confidence: 99%

Integrating Machine Learning with Human Knowledge

et al. 2020

View full text Add to dashboard Cite

Summary Machine learning has been heavily researched and widely used in many disciplines. However, achieving high accuracy requires a large amount of data that is sometimes difficult, expensive, or impractical to obtain. Integrating human knowledge into machine learning can significantly reduce data requirement, increase reliability and robustness of machine learning, and build explainable machine learning systems. This allows leveraging the vast amount of human knowledge and capability of machine learning to achieve functions and performance not available before and will facilitate the interaction between human beings and machine learning systems, making machine learning decisions understandable to humans. This paper gives an overview of the knowledge and its representations that can be integrated into machine learning and the methodology. We cover the fundamentals, current status, and recent progress of the methods, with a focus on popular and new topics. The perspectives on future directions are also discussed.

show abstract

Section: Knowledge and Its Representationsmentioning

confidence: 99%

Integrating Machine Learning with Human Knowledge

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Recently, several new stochastic approximation methods for certain classes of high-dimensional nonlinear PDEs have been proposed and studied in the scientific literature. In particular, we refer, e.g., to [11,12,26,29,30,53] for BSDE-based approximation methods for PDEs in which nested conditional expectations are discretized through suitable regression methods, we refer, e.g., to [10,39,41,42] for branching diffusion approximation methods for PDEs, we refer, e.g., to [1][2][3][6][7][8]13,14,16,17,21,24,25,31,[34][35][36]40,43,48,50,52,[54][55][56][57][58]60,62,63] for deep learning based approximation methods for PDEs, and we refer to [4,5,20,28,46,47] for numerical simulations, approximation results, and extensions of the in…”

Section: Introductionmentioning

confidence: 99%

“…For MLP approximation methods it has been recently shown in [4,45,46] that such algorithms do indeed overcome the curse of dimensionality for certain classes of gradient-independent PDEs. Numerical simulations for deep learning based approximation methods for nonlinear PDEs in high dimensions are very encouraging (see, e.g., the above named references [1][2][3][6][7][8]13,14,16,17,21,24,25,31,[34][35][36]40,43,48,50,52,[54][55][56][57][58]60,62,63]) but so far there is only partial error analysis available for such algorithms (which, in turn, is strongly based on the above-mentioned error analysis for the MLP approximation method; cf. [44] and, e.g., [9,23,32,33,36,49,51,61,62]).…”

Section: Introductionmentioning

confidence: 99%

Overcoming the Curse of Dimensionality in the Numerical Approximation of Parabolic Partial Differential Equations with Gradient-Dependent Nonlinearities

2021

View full text Add to dashboard Cite

Partial differential equations (PDEs) are a fundamental tool in the modeling of many real-world phenomena. In a number of such real-world phenomena the PDEs under consideration contain gradient-dependent nonlinearities and are high-dimensional. Such high-dimensional nonlinear PDEs can in nearly all cases not be solved explicitly, and it is one of the most challenging tasks in applied mathematics to solve high-dimensional nonlinear PDEs approximately. It is especially very challenging to design approximation algorithms for nonlinear PDEs for which one can rigorously prove that they do overcome the so-called curse of dimensionality in the sense that the number of computational operations of the approximation algorithm needed to achieve an approximation precision of size $${\varepsilon }> 0$$ ε > 0 grows at most polynomially in both the PDE dimension $$d \in \mathbb {N}$$ d ∈ N and the reciprocal of the prescribed approximation accuracy $${\varepsilon }$$ ε . In particular, to the best of our knowledge there exists no approximation algorithm in the scientific literature which has been proven to overcome the curse of dimensionality in the case of a class of nonlinear PDEs with general time horizons and gradient-dependent nonlinearities. It is the key contribution of this article to overcome this difficulty. More specifically, it is the key contribution of this article (i) to propose a new full-history recursive multilevel Picard approximation algorithm for high-dimensional nonlinear heat equations with general time horizons and gradient-dependent nonlinearities and (ii) to rigorously prove that this full-history recursive multilevel Picard approximation algorithm does indeed overcome the curse of dimensionality in the case of such nonlinear heat equations with gradient-dependent nonlinearities.

show abstract

“…The majority of recent methods for control of spatio-temporal systems typically reduce PDEs into a finite set of Ordinary Differential Equations (ODEs) through Reduced Order Models (ROMs), and apply standard finite-dimensional optimization methods which result in algorithms specific to the ROM used. Within this paradigm, deep learning methods have successfully been applied on policy networks in the finite dimensional setting for controlling Navier-Stokes systems [3,4,5,6], for soft robotic systems [7,8], as well as for many other systems [9]. These methods are often specific to a discretization scheme and represent a discretize-then-optimize approach.…”

Section: Introductionmentioning

confidence: 99%

Spatio-Temporal Differential Dynamic Programming for Control of Fields

Evans,

So,

Kendall

et al. 2021

Preprint

View full text Add to dashboard Cite

We consider the optimal control problem of a general nonlinear spatio-temporal system described by Partial Differential Equations (PDEs). Theory and algorithms for control of spatio-temporal systems are of rising interest among the automatic control community and exhibit numerous challenging characteristic from a control standpoint. Recent methods focus on finite-dimensional optimization techniques of a discretized finite dimensional ODE approximation of the infinite dimensional PDE system. In this paper, we derive a differential dynamic programming (DDP) framework for distributed and boundary control of spatio-temporal systems in infinite dimensions that is shown to generalize both the spatio-temporal LQR solution, and modern finite dimensional DDP frameworks. We analyze the convergence behavior and provide a proof of global convergence for the resulting system of continuous-time forward-backward equations. We explore and develop numerical approaches to handle sensitivities that arise during implementation, and apply the resulting STDDP algorithm to a linear and nonlinear spatio-temporal PDE system. Our framework is derived in infinite dimensional Hilbert spaces, and represents a discretization-agnostic framework for control of nonlinear spatio-temporal PDE systems.

show abstract

Deep reinforcement learning for partial differential equation control

Cited by 43 publications

References 16 publications

Integrating Machine Learning with Human Knowledge

Integrating Machine Learning with Human Knowledge

Overcoming the Curse of Dimensionality in the Numerical Approximation of Parabolic Partial Differential Equations with Gradient-Dependent Nonlinearities

Spatio-Temporal Differential Dynamic Programming for Control of Fields

Contact Info

Product

Resources

About