Critical Investigation of Failure Modes in Physics-informed Neural Networks

Abstract

In this paper, we demonstrate and investigate several challenges that stand in the way of tackling complex problems using physics-informed neural networks. In particular, we visualize the loss landscapes of trained models and perform sensitivity analysis of backpropagated gradients in the presence of physics. Our findings suggest that existing methods produce highly non-convex loss landscapes that are difficult to navigate. Furthermore, high-order PDEs contaminate the backpropagated gradients that may impede or prevent convergence. We then propose a novel method that bypasses the calculation of high-order PDE operators and mitigates the contamination of backpropagating gradients. In doing so, we reduce the dimension of the search space of our solution and facilitate learning problems with non-smooth solutions. Our formulation also provides a feedback mechanism that helps our model adaptively focus on complex regions of the domain that are difficult to learn. We then formulate an unconstrained dual problem by adapting the Lagrange multiplier method. We apply our method to solve several challenging benchmark problems governed by linear and non-linear PDEs.

Lagrangian PINNs: A causality-conforming solution to failure modes of physics-informed neural networks

“…To evaluate the optimality of the trained network, the loss landscape of the network at the end of the training phase is often used as a descriptive measure [6,10,13,52]. To compute the loss landscape, the two dominant eigenvectors of the Hessian of the loss with respect to the trainable parameters of the networks, δ and η, are computed using the code provided in [53].…”

“…However, the training phase of PINNs, equivalent to solving the PDEs, faces some challenges [6][7][8][9][10][11][12][13][14]. The innovations and attempts to improve the accuracy of the PINNs can be classified into two categories.…”

“…It is also demonstrated that the architecture of the network can meaningfully change the stiffness of the gradients in the learning phase, and therefore it is suggested that a specialized architecture can be beneficial to specific problems [9]. Subsequently, reformulating the constraints using Augmented Lagrangian method (ALM) [15] demonstrates a flatter/smoother loss landscape, and therefore leads to a more favorable training behavior [13], compared to the originally proposed penalty terms [1]. Notably, the loss landscape of PINNs are less smooth compared to purely data-driven ANNs [10,13].…”

“…Subsequently, reformulating the constraints using Augmented Lagrangian method (ALM) [15] demonstrates a flatter/smoother loss landscape, and therefore leads to a more favorable training behavior [13], compared to the originally proposed penalty terms [1]. Notably, the loss landscape of PINNs are less smooth compared to purely data-driven ANNs [10,13]. The raggedness of loss landscape explains why PINNs are more prone to converge to an unfavorable local minima.…”

“…The raggedness of loss landscape explains why PINNs are more prone to converge to an unfavorable local minima. Such challenge depends on both the governing equations, and system parameters [6,10,13].…”