A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks

Abstract

Physics-informed neural networks (PINNs) have shown to be an effective tool for solving both forward and inverse problems of partial differential equations (PDEs). PINNs embed the PDEs into the loss of the neural network using automatic differentiation, and this PDE loss is evaluated at a set of scattered spatio-temporal points (called residual points). The location and distribution of these residual points are highly important to the performance of PINNs. However, in the existing studies on PINNs, only a few simple residual point sampling methods have mainly been used. Here, we present a comprehensive study of two categories of sampling for PINNs: non-adaptive uniform sampling and adaptive nonuniform sampling. We consider six uniform sampling methods, including (1) equispaced uniform grid, (2) uniformly random sampling, (3) Latin hypercube sampling, (4) Halton sequence, (5) Hammersley sequence, and (6) Sobol sequence. We also consider a resampling strategy for uniform sampling. To improve the sampling efficiency and the accuracy of PINNs, we propose two new residual-based adaptive sampling methods: residual-based adaptive distribution (RAD) and residual-based adaptive refinement with distribution (RAR-D), which dynamically improve the distribution of residual points based on the PDE residuals during training. Hence, we have considered a total of 10 different sampling methods, including six non-adaptive uniform sampling, uniform sampling with resampling, two proposed adaptive sampling, and an existing adaptive sampling. We extensively tested the performance of these sampling methods for four forward problems and two inverse problems in many setups. Our numerical results presented in this study are summarized from more than 6000 simulations of PINNs. We show that the proposed adaptive sampling methods of RAD and RAR-D significantly improve the accuracy of PINNs with fewer residual points for both forward and inverse problems. The results obtained in this study can also be used as a practical guideline in choosing sampling methods.

Adaptive Self-Supervision Algorithms for Physics-Informed Neural Networks

“…Not only does this not increase or imbalance the computational overheard (number of data points is fixed) but it is easier to implement as well. Finally, we acknowledge two concurrent works that have appeared recently that also look at resampling strategies for PINNs [5,30] that have suggested similar ideas of dynamically resampling points through training; however they also focus only on the PDE residual for sampling.…”

DMIS: Dynamic Mesh-Based Importance Sampling for Training Physics-Informed Neural Networks

“…The training of PINNs is computationally expensive and unstable. To alleviate these defects of PINNs, allocation of loss weights (Wang, Teng, and Perdikaris 2021;Wang, Yu, and Perdikaris 2022;Krishnapriyan et al 2021), parallel computation methods (Meng et al 2020; and sampling schemes (Lu et al 2021;Nabian, Gladstone, and Meidani 2021;Daw et al 2022;Wu et al 2022) have been widely discussed and improve the convergence efficiency and prediction accuracy of PINNs. Gradient control methods are also studied (Kim et al 2021).…”

Deep Ritz method for solving high-dimensional fractional differential equations

“…Recently, various efforts have been made to exploit better point-taking techniques to increase effectiveness. The importance of point-taking methods is showed in [41]- [43]. Moreover, the results are shown in Fig.…”