A Toolkit for Data-Driven Discovery of Governing Equations in High-Noise Regimes

Abstract

We consider the data-driven discovery of governing equations from time-series data in the limit of high noise. The algorithms developed describe an extensive toolkit of methods for circumventing the deleterious effects of noise in the context of the sparse identification of nonlinear dynamics (SINDy) framework. We offer two primary contributions, both focused on noisy data acquired from a system ẋ = f (x). First, we propose, for use in high-noise settings, an extensive toolkit of critically enabling extensions for the SINDy regression method, to progressively cull functionals from an over-complete library and yield a set of sparse equations that regress to the derivate ẋ. This toolkit includes: (regression step) weight timepoints based on estimated noise, use ensembles to estimate coefficients, and regress using FFTs; (culling step) leverage linear dependence of functionals, and restore and protect culled functionals based on Figures of Merit (FoMs). In a novel Assessment step, we define FoMs that compare model predictions to the original time-series (i.e., x(t) rather than ẋ(t)). These innovations can extract sparse governing equations and coefficients from high-noise time-series data (e.g., 300% added noise). For example, it discovers the correct sparse libraries in the Lorenz system, with median coefficient estimate errors equal to 1%−3% (for 50% noise), 6%−8% (for 100% noise); and 23%−25% (for 300% noise). The enabling modules in the toolkit are combined into a single method, but the individual modules can be tactically applied in other equation discovery methods (SINDy or not) to improve results on high-noise data. Second, we propose a technique, applicable to any model discovery method based on ẋ = f (x), to assess the accuracy of a discovered model in the context of non-unique solutions due to noisy data. Currently, this non-uniqueness can obscure a discovered model's accuracy and thus a discovery method's effectiveness. We describe a technique that uses linear dependencies among functionals to transform a discovered model into an equivalent form that is closest to the true model, enabling more accurate assessment of a discovered model's accuracy.

From biological data to oscillator models using SINDy

“…3C). This high sensitivity of SINDy to noise has already been reported in the literature (40, 47, 66, 67, 81–83). To handle such noise and improve performance, different noise filtering techniques were proposed (66), and more advanced methods were developed (47, 82, 83), which we will consider later.…”

“…in Refs. (47, 63, 66), and Delahunt and Kutz have developed a toolkit for noise handling when applying the SINDy algorithm (67). For the special case of biological transport models Lagergren, Nardini et al .…”

“…in Refs. (47,63,66), and Delahunt and Kutz have developed a toolkit for noise handling when applying the SINDy algorithm (67). For the special case of biological transport models Lagergren, Nardini et al (68,69) have shown that using additional noise filtering via a neural network on (spatio)temporal data can significantly improve model identification with regression algorithms.…”

Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control

“…SINDy has been widely adopted, in part, because it is highly extensible. Extensions of the SINDy algorithm include accounting for control inputs [46] and rational functions [47,48], enforcing known conservation laws and symmetries [30], promoting stability [49], improved noise robustness through the integral formulation [37,[50][51][52][53][54], generalizations for stochastic dynamics [44,55] and tensor formulations [56], and probabilistic model discovery via sparse Bayesian inference [57][58][59][60][61]. Many of these innovations have been incorporated into the open source software package PySINDy [62,63].…”

“…This approach has been used to discover a hierarchy of PDE models for fluids and plasmas [37,[51][52][53][54]64,65]. Several works have begun to explore ensemble methods to robustify data-driven modelling, including the use of bagging for DMD [66], ensemble-Lasso [67], subsample aggregating for improved discovery [61,68], statistical learning of PDEs to select model coefficients with high importance measures [69] and improved discovery using ensembles based on subsampling of the data [51,52,61,65]. Also, symbolic regression methods [11][12][13] and spectral proper orthogonal decomposition (SPOD) [70] are inherently imbued with ensembling ideas.…”