Sparsely constrained neural networks for model discovery of PDEs

Both, Gert-Jan; Vermarien, Gijs; Kusters, Rémy

doi:10.48550/arxiv.2011.04336

Cited by 3 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Kuramoto-Shivashinksy equation describes flame propagation and is given by u t = −uu x − u xx − u xxxx . The fourth order derivative makes it challenging to learn with numerical differentiation-based methods, while its periodic and chaotic nature makes it challenging to learn with neural network based methods [8]. We show here that using the SBL-constrained approach we discover the KS-equation from only a small slice of the chaotic data (256 in space, 25 time steps), with 20% additive noise.…”

Section: Kuramoto-shivashinskymentioning

confidence: 88%

“…The mask M describes which terms make up the equation, and hence the form of the constraint; compared to PINNs, the mask M which defines the form of the constraint, is also learned. This mask is updated periodically by some sparse regression technique, and as terms are pruned, the constraint becomes stricter, preventing overfitting of the constraint itself and improving the approximation of the network, boosting performance significantly [8]. However, the non-differentiability of the mask can lead to issues during training, for example when it is updated at the wrong time, or when the wrong terms are accidentally pruned.…”

Section: Introductionmentioning

confidence: 99%

“…3 optimises two sets of parameters: the network parameters θ and the coefficients ξ. Typically both are minimised together using gradient descent, but the optimisation of ξ can be performed analytically [8]. Given a configuration of the network parameters θ * , the minimisation over ξ is a regression problem as given by eq.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Fully differentiable model discovery

Both¹,

Kusters²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Model discovery aims at autonomously discovering differential equations underlying a dataset. Approaches based on Physics Informed Neural Networks (PINNs) have shown great promise, but a fully-differentiable model which explicitly learns the equation has remained elusive. In this paper we propose such an approach by combining neural network based surrogates with Sparse Bayesian Learning (SBL). We start by reinterpreting PINNs as multitask models, applying multitask learning using uncertainty, and show that this leads to a natural framework for including Bayesian regression techniques. We then construct a robust model discovery algorithm by using SBL, which we showcase on various datasets. Concurrently, the multitask approach allows the use of probabilistic approximators, and we show a proof of concept using normalizing flows to directly learn a density model from single particle data. Our work expands PINNs to various types of neural network architectures, and connects neural network-based surrogates to the rich field of Bayesian parameter inference.Preprint. Under review.

show abstract

Section: Kuramoto-shivashinskymentioning

confidence: 88%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Fully differentiable model discovery

Both¹,

Kusters²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We can therefore expect the deterministic noise δ to be much smaller. To leverage such capability, we implement the adaptive Lasso with stability selection and error control in the deep learning model discovery framework DeepMod 4 , [20]. The framework combines a function approximator of u, typically a deep neural network which is trained with the following loss,…”

Section: Within a Deep Learning Framework To Reduce The Deterministic...mentioning

confidence: 99%

Sparsistent Model Discovery

Tod,

Both,

Kusters

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Discovering the partial differential equations underlying a spatio-temporal datasets from very limited observations is of paramount interest in many scientific fields. However, it remains an open question to know when model discovery algorithms based on sparse regression can actually recover the underlying physical processes. We trace back the poor of performance of Lasso based model discovery algorithms to its potential variable selection inconsistency: meaning that even if the true model is present in the library, it might not be selected. By first revisiting the irrepresentability condition (IRC) of the Lasso, we gain some insights of when this might occur. We then show that the adaptive Lasso will have more chances of verifying the IRC than the Lasso and propose to integrate it within a deep learning model discovery framework with stability selection and error control. Experimental results show we can recover several nonlinear and chaotic canonical PDEs with a single set of hyperparameters from a very limited number of samples at high noise levels.

show abstract

“…As pioneering researchers in sparse PDE learning, Rudy et al [1,9] modified the ridge regression method by imposing hard thresholding which recursively eliminates certain terms with coefficient values below a learned threshold. As pointed out in Limitations of [1,9] (Section 4 in Supplementary Materials) and following studies [4,10,11], the identification quality is very sensitive to data quantity and quality. For example, the terms of the reaction diffusion equation cannot be correctly identified using the data with only 0.5% random noise.…”

Section: Introductionmentioning

confidence: 99%

Robust Data-Driven Discovery of Partial Differential Equations under Uncertainties

Zhang,

Liu

2021

Preprint

View full text Add to dashboard Cite

Robust physics (e.g., governing equations and laws) discovery is of great interest for many engineering fields and explainable machine learning. A critical challenge compared with general training is that the term and format of governing equations is not known as a prior. In addition, significant measurement noise and complex algorithm hyperparameter tuning usually reduces the robustness of existing methods. A robust data-driven method is proposed in this study for identifying the governing Partial Differential Equations (PDEs) of a given system from noisy data. The proposed method is based on the concept of Progressive Sparse Identification of PDEs (PSI-PDE or ψ-PDE). Special focus is on the handling of data with huge uncertainties (e.g., 50% noise level). Neural Network modeling and fast Fourier transform (FFT) are implemented to reduce the influence of noise in sparse regression. Following this, candidate terms from the prescribed library are progressively selected and added to the learned PDEs, which automatically promotes parsimony with respect to the number of terms in PDEs as well as their complexity. Next, the significance of each learned terms is further evaluated and the coefficients of PDE terms are optimized by minimizing the L2 residuals. Results of numerical case studies indicate that the governing PDEs of many canonical dynamical systems can be correctly identified using the proposed ψ-PDE method with highly noisy data. One great benifit of proposed algorithm is that it avoids complex algorithm modification and hyperparameter tuning in most existing methods. Limitations of the proposed method and major findings are presented.

show abstract

Sparsely constrained neural networks for model discovery of PDEs

Cited by 3 publications

References 14 publications

Fully differentiable model discovery

Fully differentiable model discovery

Sparsistent Model Discovery

Robust Data-Driven Discovery of Partial Differential Equations under Uncertainties

Contact Info

Product

Resources

About