Connections between deep learning and partial differential equations

Burger, Martin; E, W.; Ruthotto, Lars; Osher, S.

doi:10.1017/s0956792521000085

Cited by 8 publications

(3 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[13] is more focused on generalization rather than numerical stability and does not address restrained instabilites. More broadly, although not directly related to our work, there has been a number of recent work in connecting PDEs with neural networks for various end goals (see [14]- [16] for detailed surveys). For example, applying neural networks to solve PDEs [17]- [21], and interpreting the forward pass of neural networks as approximations to PDEs [22]- [24].…”

Section: Related Workmentioning

confidence: 99%

Surprising Instabilities in Training Deep Networks and a Theoretical Analysis

Sun¹,

Lao²,

Sundaramoorthi³

et al. 2022

Preprint

View full text Add to dashboard Cite

We discover restrained numerical instabilities in current training practices of deep networks with SGD. We show numerical error (on the order of the smallest floating point bit) induced from floating point arithmetic in training deep nets can be amplified significantly and result in significant test accuracy variance, comparable to the test accuracy variance due to stochasticity in SGD. We show how this is likely traced to instabilities of the optimization dynamics that are restrained, i.e., localized over iterations and regions of the weight tensor space. We do this by presenting a theoretical framework using numerical analysis of partial differential equations (PDE), and analyzing the gradient descent PDE of a simplified convolutional neural network (CNN). We show that it is stable only under certain conditions on the learning rate and weight decay. We reproduce the localized instabilities in the PDE for the simplified network, which arise when the conditions are violated.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Surprising Instabilities in Training Deep Networks and a Theoretical Analysis

Sun¹,

Lao²,

Sundaramoorthi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In the last two decades, machine learning and deep learning techniques have started to play an active role in the setting up of new methods for the numerical solution of PDEs [22][23][24]. In particular, Physics-Informed Neural Networks (PINNs) [25][26][27] have emerged as an intuitive and efficient deep learning framework to solve PDEs, carrying on the training of a neural network by minimizing the loss functional which incorporates the PDE itself, informing the neural network about the physical problem to be solved.…”

Section: Introductionmentioning

confidence: 99%

Splines Parameterization of Planar Domains by Physics-Informed Neural Networks

et al. 2023

View full text Add to dashboard Cite

The generation of structured grids on bounded domains is a crucial issue in the development of numerical models for solving differential problems. In particular, the representation of the given computational domain through a regular parameterization allows us to define a univalent mapping, which can be computed as the solution of an elliptic problem, equipped with suitable Dirichlet boundary conditions. In recent years, Physics-Informed Neural Networks (PINNs) have been proved to be a powerful tool to compute the solution of Partial Differential Equations (PDEs) replacing standard numerical models, based on Finite Element Methods and Finite Differences, with deep neural networks; PINNs can be used for predicting the values on simulation grids of different resolutions without the need to be retrained. In this work, we exploit the PINN model in order to solve the PDE associated to the differential problem of the parameterization on both convex and non-convex planar domains, for which the describing PDE is known. The final continuous model is then provided by applying a Hermite type quasi-interpolation operator, which can guarantee the desired smoothness of the sought parameterization. Finally, some numerical examples are presented, which show that the PINNs-based approach is robust. Indeed, the produced mapping does not exhibit folding or self-intersection at the interior of the domain and, also, for highly non convex shapes, despite few faulty points near the boundaries, has better shape-measures, e.g., lower values of the Winslow functional.

show abstract

“…In the context of surrogate modeling and uncertainty quantification (UQ), DNN methods include Bayesian deep convolutional encoder-decoder networks [71], deep multi-scale model learning [63], physics-constrained deep learning method [72], see also [26,55,25,68] and references therein. For DNN applications in mean field games (high dimensional optimal control problems) and various connections with numerical PDEs, see [6,7,53,3,35] and references therein. In view of the above literature, the DNN interactions with numerical PDE methods mostly occur in the Eulerian setting with PDE solutions defined in either strong or weak (variational) sense.…”

Section: Introductionmentioning

confidence: 99%

DeepParticle: learning invariant measure by a deep neural network minimizing Wasserstein distance on data generated from an interacting particle method

Wang¹,

Xin²,

Zhang³

2021

Preprint

View full text Add to dashboard Cite

We introduce DeepParticle, a method to learn and generate invariant measures of stochastic dynamical systems with physical parameters based on data computed from an interacting particle method (IPM). We utilize the expressiveness of deep neural networks (DNNs) to represent the transform of samples from a given input (source) distribution to an arbitrary target distribution, neither assuming distribution functions in closed form nor the sample transforms to be invertible, nor the sample state space to be finite. In the training stage, we update the weights of the network by minimizing a discrete Wasserstein distance between the input and target samples. To reduce the computational cost, we propose an iterative divide-and-conquer (a mini-batch interior point) algorithm, to find the optimal transition matrix in the Wasserstein distance. We present numerical results to demonstrate the performance of our method for accelerating the computation of invariant measures of stochastic dynamical systems arising in computing reaction-diffusion front speeds in 3D chaotic flows by using the IPM. The physical parameter is a large Péclet number reflecting the advection-dominated regime of our interest.

show abstract

Connections between deep learning and partial differential equations

Cited by 8 publications

References 7 publications

Surprising Instabilities in Training Deep Networks and a Theoretical Analysis

Surprising Instabilities in Training Deep Networks and a Theoretical Analysis

Splines Parameterization of Planar Domains by Physics-Informed Neural Networks

DeepParticle: learning invariant measure by a deep neural network minimizing Wasserstein distance on data generated from an interacting particle method

Contact Info

Product

Resources

About