Stable Neural Flows

Massaroli, Stefano; Poli, Michael; Bin, Michelangelo; Park, Jinkyoo; Yamashita, Atsushi; Asama, Hajime

doi:10.48550/arxiv.2003.08063

Cited by 20 publications

(28 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our approach, which relies on using the energy as a Lyapunov function for an entire class of models with fixed nonlinear structure, is challenging for application to higher-order nonlinearities where generic Lyapunov functions are often unknown. Fortunately, data-driven methods are now increasingly used to discover Lyapunov functions and barrier functions for nonlinear control [91,[119][120][121][122][123][124][125][126]. These methods build a heuristic Lyapunov function for a given dataset, rendering the search for a Lyapunov function tractable but possibly at the cost of model generality.…”

Section: Discussionmentioning

confidence: 99%

Promoting global stability in data-driven models of quadratic nonlinear dynamics

Kaptanoglu,

Callaham,

Hansen

et al. 2021

Preprint

View full text Add to dashboard Cite

Modeling realistic fluid and plasma flows is computationally intensive, motivating the use of reducedorder models for a variety of scientific and engineering tasks. However, it is challenging to characterize, much less guarantee, the global stability (i.e., long-time boundedness) of these models. The seminal work of Schlegel and Noack [1] provided a theorem outlining necessary and sufficient conditions to ensure global stability in systems with energy-preserving, quadratic nonlinearities, with the goal of evaluating the stability of projection-based models. In this work, we incorporate this theorem into modern data-driven models obtained via machine learning. First, we propose that this theorem should be a standard diagnostic for the stability of projection-based and data-driven models, examining the conditions under which it holds. Second, we illustrate how to modify the objective function in machine learning algorithms to promote globally stable models, with implications for the modeling of fluid and plasma flows. Specifically, we introduce a modified "trapping SINDy" algorithm based on the sparse identification of nonlinear dynamics (SINDy) method. This method enables the identification of models that, by construction, only produce bounded trajectories. The effectiveness and accuracy of this approach are demonstrated on a broad set of examples of varying model complexity and physical origin, including the vortex shedding in the wake of a circular cylinder.

show abstract

Section: Discussionmentioning

confidence: 99%

Promoting global stability in data-driven models of quadratic nonlinear dynamics

Kaptanoglu,

Callaham,

Hansen

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…As the complexity or the dimensionality of the modeling task increases, ODE-based networks demand a more advanced solver that significantly impacts their efficiency (Poli et al, 2020), stability (Bai et al, 2019, Chang et al, 2019, Lechner et al, 2020b, Massaroli et al, 2020a and performance . A large body of research went into improving the computational overhead of these solvers, for example, by designing hypersolvers (Poli et al, 2020), deploying augmentation methods (Dupont et al, 2019, Massaroli et al, 2020b, pruning (Liebenwein et al, 2021) and by regularizing the continuous flows (Finlay et al, 2020, Kidger et al, 2020, Massaroli et al, 2020a. To enhance the performance of an ODE-based model, especially in time series modeling tasks (Gleeson et al, 2018), solutions provided for stabilizing their gradient propagation (Erichson et al, 2021, Lechner and Hasani, 2020, Li et al, 2020.…”

Section: Related Workmentioning

confidence: 99%

“…The community has worked out solutions for resolving this computational overhead and facilitating the training of neural ODEs, for instance, by relaxing the stiffness of a flow by state augmentation techniques (Dupont et al, 2019, Massaroli et al, 2020b, reformulating the forward-pass as a rootfinding problem (Bai et al, 2019), using regularization schemes (Finlay et al, 2020, Kidger et al, 2020, Massaroli et al, 2020a, or improving the inference time of the network (Poli et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

Closed-form Continuous-time Neural Models

Hasani¹,

Lechner²,

Amini³

et al. 2021

Preprint

View full text Add to dashboard Cite

Continuous-depth neural models, where the derivative of the model's hidden state is defined by a neural network, have enabled strong sequential data processing capabilities. However, these models rely on advanced numerical differential equation (DE) solvers resulting in a significant overhead both in terms of computational cost and model complexity. In this paper, we present a new family of models, termed Closed-form Continuous-depth (CfC) networks, that are simple to describe and at least one order of magnitude faster while exhibiting equally strong modeling abilities compared to their ODE-based counterparts. The models are hereby derived from the analytical closed-form solution of an expressive subset of time-continuous models, thus alleviating the need for complex DE solvers all together. In our experimental evaluations, we demonstrate that CfC networks outperform advanced, recurrent models over a diverse set of time-series prediction tasks, including those with long-term dependencies and irregularly sampled data. We believe our findings open new opportunities to train and deploy rich, continuous neural models in resource-constrained settings, which demand both performance and efficiency.

show abstract

“…By using adjoint method [5],the gradients w.r.t.θ can be computed memory-efficiently during back propagation. Recent works [10,26,7,38,12] have tried to analyze this framework theoretically, overcome the instability issue and improve memory-efficiency. Researchers also pay attention to apply NODE to other fields such as medical image [28], reinforcement learning [9], video generation [17,41] and graph data [37].…”

Section: Background and Related Workmentioning

confidence: 99%

STR-GODEs: Spatial-Temporal-Ridership Graph ODEs for Metro Ridership Prediction

Huang

2021

Preprint

View full text Add to dashboard Cite

The metro ridership prediction has always received extensive attention from governments and researchers. Recent works focus on designing complicated graph convolutional recurrent network architectures to capture spatial and temporal patterns. These works extract the information of spatial dimension well, but the limitation of temporal dimension still exists. We extended Neural ODE algorithms to the graph network and proposed the STR-GODEs network, which can effectively learn spatial, temporal, and ridership correlations without the limitation of dividing data into equal-sized intervals on the timeline. While learning the spatial relations and the temporal correlations, we modify the GODE-RNN cell to obtain the ridership feature and hidden states. Ridership information and its hidden states are added to the GODESolve to reduce the error accumulation caused by long time series in prediction. Extensive experiments on two large-scale datasets demonstrate the efficacy and robustness of our model. 1

show abstract

Stable Neural Flows

Cited by 20 publications

References 0 publications

Promoting global stability in data-driven models of quadratic nonlinear dynamics

Promoting global stability in data-driven models of quadratic nonlinear dynamics

Closed-form Continuous-time Neural Models

STR-GODEs: Spatial-Temporal-Ridership Graph ODEs for Metro Ridership Prediction

Contact Info

Product

Resources

About