Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics

Abstract

We address a three-tier numerical framework based on nonlinear manifold learning for the forecasting of high-dimensional time series, relaxing the “curse of dimensionality” related to the training phase of surrogate/machine learning models. At the first step, we embed the high-dimensional time series into a reduced low-dimensional space using nonlinear manifold learning (local linear embedding and parsimonious diffusion maps). Then, we construct reduced-order surrogate models on the manifold (here, for our illustrations, we used multivariate autoregressive and Gaussian process regression models) to forecast the embedded dynamics. Finally, we solve the pre-image problem, thus lifting the embedded time series back to the original high-dimensional space using radial basis function interpolation and geometric harmonics. The proposed numerical data-driven scheme can also be applied as a reduced-order model procedure for the numerical solution/propagation of the (transient) dynamics of partial differential equations (PDEs). We assess the performance of the proposed scheme via three different families of problems: (a) the forecasting of synthetic time series generated by three simplistic linear and weakly nonlinear stochastic models resembling electroencephalography signals, (b) the prediction/propagation of the solution profiles of a linear parabolic PDE and the Brusselator model (a set of two nonlinear parabolic PDEs), and (c) the forecasting of a real-world data set containing daily time series of ten key foreign exchange rates spanning the time period 3 September 2001–29 October 2020.

Some recent trends in embeddings of time series and dynamic networks

“…This has been examined by for example Papaioannou et al . (2021). They use locally linear embedding (LLE) and diffusion maps as their two embeddings routines, and with radial basis function interpolation and geometric harmonics to create back‐transformations.…”

Data-driven Control of Agent-based Models: an Equation/Variable-free Machine Learning Approach

“…Below, we present the two methodologies utilized in this work to solve the pre-image problem, namely the Geometric Harmonics (GHs) and the k-Nearest Neighbors (k-NN) algorithms. For a detailed review and comparison of various methods see [9,18,65].…”

“…Scientific computation and control of the emergent/collective dynamics of high-dimensional multiscale/complex dynamical systems constitute open challenging tasks due to (a) the lack of physical insight and knowledge of the appropriate macroscopic quantities needed to usefully describe the evolution of the emergent dynamics, (b) the so-called "curse of dimensionality" when trying to efficiently learn surrogate models with good generalization properties, and (c) the problem of bridging the scale where individual units (atoms, molecules, cells, bacteria, individuals, robots) interact, and the macroscopic scale where the emergent properties arise and evolve [1][2][3][4]. For the task of identification of macroscopic variables from high-fidelity simulations/spatio-temporal data, various machine learning methods have been proposed including non-linear manifold learning algorithms such as Diffusion Maps (DMs) [5][6][7][8][9][10][11][12][13], ISOMAP [14][15][16] and Local Linear Embedding [17,18] but also Autoencoders [19,20]. For the task of the extraction of surrogate models for the approximation of the emergent dynamics, available approaches include the Sparse Identification of the Nonlinear Dynamics (SINDy) [21], the Koopman operator [22][23][24][25][26][27], Gaussian Processes [12,18,28], Artificial Neural Networks (ANNs) [12,13], Recursive Neural Networks (RNN) [20], Deep Learning [29], as well as Long Short-Term Memory (LSTM) networks [30].…”

“…For the task of identification of macroscopic variables from high-fidelity simulations/spatio-temporal data, various machine learning methods have been proposed including non-linear manifold learning algorithms such as Diffusion Maps (DMs) [5][6][7][8][9][10][11][12][13], ISOMAP [14][15][16] and Local Linear Embedding [17,18] but also Autoencoders [19,20]. For the task of the extraction of surrogate models for the approximation of the emergent dynamics, available approaches include the Sparse Identification of the Nonlinear Dynamics (SINDy) [21], the Koopman operator [22][23][24][25][26][27], Gaussian Processes [12,18,28], Artificial Neural Networks (ANNs) [12,13], Recursive Neural Networks (RNN) [20], Deep Learning [29], as well as Long Short-Term Memory (LSTM) networks [30]. For the task of the bridging of the micro-and macro-scales, the Equation-free (EF) multiscale framework, introduced back in the early 2000s years, [1,31,32] bypasses the need to extract explicit, closed form macroscopic/surrogate models of any type (e.g.…”

Task-oriented machine learning surrogates for tipping points of agent-based models

“…Here, based on our previous efforts on the construction of latent spaces 52 , 54 , 55 and ROM surrogates via machine learning (ML) 29 , 30 , 34 , 45 , 46 from microscopic detailed spatio-temporal simulations, we present an integrated ML framework for the construction of two types of surrogate models: global as well as local. In particular, we learn (a) mesoscopic Integro Partial Differential Equations (IPDEs), and (b)—guided by the EF framework 1 , 56 —local embedded low-dimensional mean-field Stochastic Differential Equations (SDEs), for the detection of tipping points and the construction of the probability distribution of the catastrophic transitions that occur in their neighborhood.…”