Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint

Cyr, Eric C; Gulian, Mamikon; Patel, Ravi G.; Perego, Mauro; Trask, Nathaniel

doi:10.48550/arxiv.1912.04862

Cited by 6 publications

(12 citation statements)

References 16 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While we have presented here a number of techniques to obtain qualitatively correct and physically meaningful solutions, the barrier in achieving convergence of error with respect to neural network size remains a major challenge to obtaining DNN solutions competitive with traditional finite element/volume methods. We refer the interested reader to some of our ongoing work in this area [75].…”

Section: Discussionmentioning

confidence: 99%

Thermodynamically consistent physics-informed neural networks for hyperbolic systems

Patel¹,

Manickam²,

Trask³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Physics-informed neural network architectures have emerged as a powerful tool for developing flexible PDE solvers which easily assimilate data, but face challenges related to the PDE discretization underpinning them. By instead adapting a least squares space-time control volume scheme, we circumvent issues particularly related to imposition of boundary conditions and conservation while reducing solution regularity requirements. Additionally, connections to classical finite volume methods allows application of biases toward entropy solutions and total variation diminishing properties. For inverse problems, we may impose further thermodynamic biases, allowing us to fit shock hydrodynamics models to molecular simulation of rarefied gases and metals. The resulting data-driven equations of state may be incorporated into traditional shock hydrodynamics codes.

show abstract

Section: Discussionmentioning

confidence: 99%

Thermodynamically consistent physics-informed neural networks for hyperbolic systems

Patel¹,

Manickam²,

Trask³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The standard approach to computing optima of objective functions associated with neural networks is to apply a gradient-based optimizer to all of the parameters in θ, for example [1,6,14,16,21,23,24]. In order to solve (2.16), we instead propose a training procedure in the spirit of [4] as follows:…”

Section: Neural Network Approximation Of Augmented Basis Function In ...mentioning

confidence: 99%

“…The learning rate for each basis function ϕ i is α i = 1×10 −2 1.1 i−1 . The hidden parameters are initialized according to the box initialization in [4]. We employ fixed tensor product Gauss-Legendre quadrature rule with 100×100 nodes in order to approximate inner products in the interior of the domain.…”

Section: Beam With Applied Couplementioning

confidence: 99%

Galerkin Neural Networks: A Framework for Approximating Variational Equations with Error Control

Ainsworth¹,

Dong²

2021

Preprint

View full text Add to dashboard Cite

We present a new approach to using neural networks to approximate the solutions of variational equations, based on the adaptive construction of a sequence of finite-dimensional subspaces whose basis functions are realizations of a sequence of neural networks. The finite-dimensional subspaces are then used to define a standard Galerkin approximation of the variational equation. This approach enjoys a number of advantages, including: the sequential nature of the algorithm offers a systematic approach to enhancing the accuracy of a given approximation; the sequential enhancements provide a useful indicator for the error that can be used as a criterion for terminating the sequential updates; the basic approach is largely oblivious to the nature of the partial differential equation under consideration; and, some basic theoretical results are presented regarding the convergence (or otherwise) of the method which are used to formulate basic guidelines for applying the method.

show abstract

“…However, it may also have different domain and image dimensionality based on the structure of network [31,30]. An adaptive basis viewpoint of DNNs is also given in [40].…”

Section: Hp-variational Physics-informed Neural Network (Hp-vpinn)mentioning

confidence: 99%

hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition

Kharazmi,

Zhang,

Karniadakis

2020

Preprint

View full text Add to dashboard Cite

We formulate a general framework for hp-variational physics-informed neural networks (hp-VPINNs) based on the nonlinear approximation of shallow and deep neural networks and hp-refinement via domain decomposition and projection onto space of high-order polynomials. The trial space is the space of neural network, which is defined globally over the whole computational domain, while the test space contains the piecewise polynomials. Specifically in this study, the hp-refinement corresponds to a global approximation with local learning algorithm that can efficiently localize the network parameter optimization. We demonstrate the advantages of hp-VPINNs in accuracy and training cost for several numerical examples of function approximation and solving differential equations.

show abstract

Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint

Cited by 6 publications

References 16 publications

Thermodynamically consistent physics-informed neural networks for hyperbolic systems

Thermodynamically consistent physics-informed neural networks for hyperbolic systems

Galerkin Neural Networks: A Framework for Approximating Variational Equations with Error Control

hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition

Contact Info

Product

Resources

About