Universal Differential Equations for Scientific Machine Learning

Abstract

In the context of science, the well-known adage “a picture is worth a thousand words” might well be “a model is worth a thousand datasets.” Scientific models, such as Newtonian physics or biological gene regulatory networks, are human-driven simplifications of complex phenomena that serve as surrogates for the countless experiments that validated the models. Recently, machine learning has been able to overcome the inaccuracies of approximate modeling by directly learning the entire set of nonlinear interactions from data. However, without any predetermined structure from the scientific basis behind the problem, machine learning approaches are flexible but data-expensive, requiring large databases of homogeneous labeled training data. A central challenge is reconciling data that is at odds with simplified models without requiring "big data". In this work demonstrate how a mathematical object, which we denote universal differential equations (UDEs), can be utilized as a theoretical underpinning to a diverse array of problems in scientific machine learning to yield efficient algorithms and generalized approaches. The UDE model augments scientific models with machine-learnable structures for scientifically-based learning. We show how UDEs can be utilized to discover previously unknown governing equations, accurately extrapolate beyond the original data, and accelerate model simulation, all in a time and data-efficient manner. This advance is coupled with open-source software that allows for training UDEs which incorporate physical constraints, delayed interactions, implicitly-defined events, and intrinsic stochasticity in the model. Our examples show how a diverse set of computationally-difficult modeling issues across scientific disciplines, from automatically discovering biological mechanisms to accelerating the training of physics-informed neural networks and large-eddy simulations, can all be transformed into UDE training problems that are efficiently solved by a single software methodology.

A Machine Learning-Aided Global Diagnostic and Comparative Tool to Assess Effect of Quarantine Control in COVID-19 Spread

“…Recently, it has been shown that neural networks can be used as function approximators to recover unknown constitutive relationships in a system of coupled ODEs. 52 , 54 Following this principle, we represent as an n layer-deep neural network with weights , activation function r , and the input vector as …”

“…For most of the regions under consideration, were optimized by minimizing the loss function given in ( Equation 13 ). Minimization was employed using local adjoint sensitivity analysis 54 , 68 following a similar procedure outlined in a recent study 52 with the ADAM optimizer, 69 with a learning rate of 0.01. The iterations required for convergence varied based on the region considered and generally ranged from 40,000 to 100,000.…”

“…The downside is that the presence of the neural network term as a component of the ODEs results in limited interpretability. The recently emerging field of scientific machine learning 51 exploits conservation principles within a universal differential equation, 52 SIR in our case, to mitigate overfitting and other related machine learning risks.…”

Hybrid Mechanistic + Neural Model of Laboratory Helicopter

“…Based on the ideas in (Chen et al, 2018), a set of packages for computer language Julia (Bezanson et al, 2017) are combined to fit neural differential equation models (Rackauckas et al, 2019(Rackauckas et al, , 2020 to experimental data, leading to differential equations that can be solved by standard differential equation solvers (Rackauckas and Nie, 2017). However, the packages aim higher: they allow for a very general mixing of mechanistic models and neural differential equation models in the same framework, with possibilities for the user to choose whether only parameters in the neural network model are tuned, or parameters in both the mechanistic model and the neural differential equation.…”

Differentiable molecular simulation can learn all the parameters in a coarse-grained force field for proteins

“…The idea of differentiating through the numerical solution of Newton’s equations of motion to obtain gradients that can be used to improve a learned force field has been discussed [28], but is yet to produce generally useful force fields. Conceptually the idea is related to recent work on neural differential equations [29, 30]. The idea is appealing because a large number of parameters can be improved at once, rather than the small numbers currently modified.…”