Universal Differential Equations for Scientific Machine Learning

Abstract

In the context of science, the well-known adage "a picture is worth a thousand words" might well be "a model is worth a thousand datasets." Scientific models, such as Newtonian physics or biological gene regulatory networks, are human-driven simplifications of complex phenomena that serve as surrogates for the countless experiments that validated the models. Recently, machine learning has been able to overcome the inaccuracies of approximate modeling by directly learning the entire set of nonlinear interactions from data. However, without any predetermined structure from the scientific basis behind the problem, machine learning approaches are flexible but data-expensive, requiring large databases of homogeneous labeled training data. A central challenge is reconciling data that is at odds with simplified models without requiring "big data". In this work we develop a new methodology, universal differential equations (UDEs), which augments scientific models with machinelearnable structures for scientifically-based learning. We show howUDEs can be utilized to discover previously unknown governing equations, accurately extrapolate beyond the original data, and accelerate model simulation, all in a time and data-efficient manner. This advance is coupled with open-source software that allows for training UDEs which incorporate physical constraints, delayed interactions, implicitly-defined events, and intrinsic stochasticity in the model. Our examples show how a diverse set of computationallydifficult modeling issues across scientific disciplines, from automatically discovering biological mechanisms to accelerating climate simulations by 15,000x, can be handled by training UDEs.

A neural ordinary differential equation framework for modeling inelastic stress response via internal state variables

“…This expansion of the state space of the dynamical system enabled learning complex dynamics with simpler flows, apparently generalized better, achieved lower losses with fewer parameters, and was more stable in training. In 2020 Rackauckas et al [43] developed a similar framework, called Universal Differential Equations (UDEs), based on universal approximation properties and focused on physical applications. In fact, NODEs and the like have been shown to have a universal approximation property [44][45][46], like the related MLPs [47,48].…”

Neural Network Verification in Control

“…• learning a control policy [37]- [39] • learning the system's model from data [40], possibly using physical constraints [41]- [44] • online estimation of states or parameters [45], [46] • learning a representation for verification [47]- [50] Despite this progress, NNs still present many challenges for control systems. For example, deep learning algorithms typically require massive amounts of training data, which can be expensive or dangerous to acquire and annotate.…”

Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control

“…Data-driven model discovery enables the characterization of complex systems where first principles derivations remain elusive, such as in neuroscience, power grids, epidemiology, finance, and ecology. A wide range of data-driven model discovery methods exist, including equation-free modeling [1], normal form identification [2][3][4], nonlinear Laplacian spectral analysis [5], Koopman analysis [6,7] and dynamic mode decomposition (DMD) [8][9][10], symbolic regression [11][12][13][14][15], sparse regression [16,17], Gaussian processes [18], and deep learning [19][20][21][22][23][24]. Limited data and noisy measurements are fundamental challenges for all of these model discovery methods, often limiting the effectiveness of such techniques across diverse application areas.…”