Model Inference for Ordinary Differential Equations by Parametric Polynomial Kernel Regression

Abstract

Model inference for dynamical systems aims to estimate the future behaviour of a system from observations. Purely model-free statistical methods, such as Artificial Neural Networks, tend to perform poorly for such tasks. They are therefore not well suited to many questions from applications, for example in Bayesian filtering and reliability estimation.This work introduces a parametric polynomial kernel method that can be used for inferring the future behaviour of Ordinary Differential Equation models, including chaotic dynamical systems, from observations. Using numerical integration techniques, parametric representations of Ordinary Differential Equations can be learnt using Backpropagation and Stochastic Gradient Descent. The polynomial technique presented here is based on a nonparametric method, kernel ridge regression. However, the time complexity of nonparametric kernel ridge regression scales cubically with the number of training data points. Our parametric polynomial method avoids this manifestation of the curse of dimensionality, which becomes particularly relevant when working with large time series data sets.Two numerical demonstrations are presented. First, a simple regression test case is used to illustrate the method and to compare the performance with standard Artificial Neural Network techniques. Second, a more substantial test case is the inference of a chaotic spatio-temporal dynamical system, the Lorenz-Emanuel system, from observations. Our method was able to successfully track the future behaviour of the system over time periods much larger than the training data sampling rate. Finally, some limitations of the method are presented, as well as proposed directions for future work to mitigate these limitations.

Probabilistic solution of chaotic dynamical system inverse problems using Bayesian Artificial Neural Networks

“…Artificial Neural Networks (ANNs) (composed of compute graphs, see [3,10]) are able to represent nonlinear functions by weighted composition of simpler functions. Roughly, a neural network architecture is defined by a directed graph, consisting of a set of nodes and edges.…”

“…Roughly, a neural network architecture is defined by a directed graph, consisting of a set of nodes and edges. A complete definition is provided in [10]. The power of compute graph representation of functions is that a large number of possible alternative function choices can be searched efficiently.…”

“…Following the notation in §4.1 of [10], we consider in this paper continuoustime dynamical systems that can be expressed as coupled first-order Ordinary Differential Equations (ODEs) of the form…”

“…The predictive Bayesian network computing f can be found by training a Bayesian compute graph on approximations to the time derivatives of u t . In [10] a method for approximating f θ given an integral discretisation was used.…”

“…For the purposes of this task, the latent system model is assumed to be a polynomial function of the observable variables. As such, an appropriate choice of architecture is a parametric polynomial kernel (see [10]). The parametric polynomial kernel allows for a polynomial structure to be assumed, even if the particular polynomial is unknown.…”