Respecting causality is all you need for training physics-informed neural networks

Abstract

While the popularity of physics-informed neural networks (PINNs) is steadily rising, to this date PINNs have not been successful in simulating dynamical systems whose solution exhibits multi-scale, chaotic or turbulent behavior. In this work we attribute this shortcoming to the inability of existing PINNs formulations to respect the spatio-temporal causal structure that is inherent to the evolution of physical systems. We argue that this is a fundamental limitation and a key source of error that can ultimately steer PINN models to converge towards erroneous solutions. We address this pathology by proposing a simple re-formulation of PINNs loss functions that can explicitly account for physical causality during model training. We demonstrate that this simple modification alone is enough to introduce significant accuracy improvements, as well as a practical quantitative mechanism for assessing the convergence of a PINNs model. We provide state-of-the-art numerical results across a series of benchmarks for which existing PINNs formulations fail, including the chaotic Lorenz system, the Kuramoto-Sivashinsky equation in the chaotic regime, and the Navier-Stokes equations in the turbulent regime. To the best of our knowledge, this is the first time that PINNs have been successful in simulating such systems, introducing new opportunities for their applicability to problems of industrial complexity.

Improved Training of Physics-Informed Neural Networks with Model Ensembles

“…The closest to our approach is the idea of splitting the solution interval into multiple sub-intervals and fitting PINNs on each sub-interval separately [10]. Another related method [22] proposes adaptive weights for the individual points that contribute to the PDE loss (6). The point weights are calculated based on the cumulative PDE error for the preceding points.…”

“…It has been noted by many practitioners (see, e.g., [10,19,22]) that solving PINNs on too large an interval often results in convergence to a bad solution. The authors of [10] advocate that the reason for this behavior is the difficulty of the optimization problem when the solution of a differential equation has a complex shape.…”

Predicting parametric spatiotemporal dynamics by multi-resolution PDE structure-preserved deep learning

“…The merits of PINNs have been demonstrated over various scientific applications, including fast surrogate/meta modeling [18][19][20], parameter/field inversion [21][22][23][24], and solving high-dimensional PDEs [25][26][27], to name a few. Due to the scalability challenges of the pointwise fully-connected PINN formulation to learn continuous functions [28][29][30][31] or operators [32][33][34], many remedies and improvements in terms of training and convergence have been proposed [35][36][37]. In particular, there is a growing trend in developing field-to-field discrete PINNs by leveraging convolution operations and numerical discretizations, which have been demonstrated to be more efficient in spatiotemporal learning [38,39].…”

Lagrangian PINNs: A causality-conforming solution to failure modes of physics-informed neural networks

“…However, the training phase of PINNs, equivalent to solving the PDEs, faces some challenges [6][7][8][9][10][11][12][13][14]. The innovations and attempts to improve the accuracy of the PINNs can be classified into two categories.…”

“…Such experiments demonstrate that the used architectures are expressive enough, and therefore, the challenge lies in the training regime. It is argued that all such challenges can be more naturally overcome by respecting the underlying spatio-temporal causality [12]. One natural approach to impose such causality is by prioritizing the earlier time steps in the training phase [12].…”