Convolutional Neural Networks combined with Runge-Kutta Methods

Zhu, Mai; Chang, Bo; Fu, Chong

doi:10.48550/arxiv.1802.08831

Cited by 15 publications

(17 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, we found that dense layers with a layer combination are better. As reported in the previous section, our best accuracy was all achieved by RK4, which internally constructs connections similar to DenseNet or FractalNet as described in Table 1 [26,28,47]. This is well aligned with the observation in ODEs that the explicit Euler method is inferior to RK4 in solve integral problems.…”

Section: Discussion On Linear Vs Densesupporting

confidence: 77%

“…𝜽 𝑓 ), and 𝑓 4 = 𝑓 (𝒉(𝑡) + 𝑠 𝑓 3 , 𝑡 + 𝑠; 𝜽 𝑓 ). It is also known that dense convolutional networks (DenseNets [47]) and fractal neural networks (FractalNet [26]) are similar to RK4 (as so are residual networks to the explicit Euler method) [28]. For simplicity but without loss of generality, however, we use the explicit Euler method as our running example.…”

Section: Residual/dense Connections and Ode Solversmentioning

confidence: 99%

See 1 more Smart Citation

LT-OCF: Learnable-Time ODE-based Collaborative Filtering

Choi,

Jeon,

Park

2021

Preprint

View full text Add to dashboard Cite

Collaborative filtering (CF) is a long-standing problem of recommender systems. Many novel methods have been proposed, ranging from classical matrix factorization to recent graph convolutional network-based approaches. After recent fierce debates, researchers started to focus on linear graph convolutional networks (GCNs) with a layer combination, which show state-of-the-art accuracy in many datasets. In this work, we extend them based on neural ordinary differential equations (NODEs), because the linear GCN concept can be interpreted as a differential equation, and present the method of Learnable-Time ODE-based Collaborative Filtering (LT-OCF). The main novelty in our method is that after redesigning linear GCNs on top of the NODE regime, i) we learn the optimal architecture rather than relying on manually designed ones, ii) we learn smooth ODE solutions that are considered suitable for CF, and iii) we test with various ODE solvers that internally build a diverse set of neural network connections. We also present a novel training method specialized to our method. In our experiments with three benchmark datasets, our method consistently outperforms existing methods in terms of various evaluation metrics. One more important discovery is that our best accuracy was achieved by dense connections. CCS CONCEPTS• Computing methodologies → Machine learning; Neural networks.

show abstract

Section: Discussion On Linear Vs Densesupporting

confidence: 77%

Section: Residual/dense Connections and Ode Solversmentioning

confidence: 99%

LT-OCF: Learnable-Time ODE-based Collaborative Filtering

Choi,

Jeon,

Park

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…[16,5,47] introduce non-regression losses inspired by Hamiltonian mechanics [18]. [38,12] have designed specific architectures for predicting and identifying dynamical systems inspired by numerical schemes for solving PDEs and residual neural networks [4,33,26,60]. [11] propose the separation of variables as a general paradigm based on a resolution method for partial differential equations for video prediction and disentanglement.…”

Section: Related Workmentioning

confidence: 99%

TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction

Pourheydari¹,

Fayyaz²,

Bahrami³

et al. 2021

Preprint

View full text Add to dashboard Cite

While recurrent neural networks (RNNs) demonstrate outstanding capabilities in future video frame prediction, they model dynamics in a discrete time space and sequentially go through all frames until the desired future temporal step is reached. RNNs are therefore prone to accumulate the error as the number of future frames increases. In contrast, partial differential equations (PDEs) model physical phenomena like dynamics in continuous time space, however, current PDE-based approaches discretize the PDEs using e.g., the forward Euler method. In this work, we therefore propose to approximate the motion in a video by a continuous function using the Taylor series. To this end, we introduce TayloSwiftNet, a novel convolutional neural network that learns to estimate the higher order terms of the Taylor series for a given input video. TayloSwiftNet can swiftly predict any desired future frame in just one forward pass and change the temporal resolution on-the-fly. The experimental results on various datasets demonstrate the superiority of our model.

show abstract

“…The new architecture was used to achieve higher accuracy on image classification. Zhu [30] introduced Runge-Kutta method to build a convolutional neural network and achieved superior accuracy. Chen [31] demonstrated the neural ordinary differential equations in continuous-depth residual networks and continuous-time latent variable models.…”

Section: Background and Related Workmentioning

confidence: 99%

Time-Continuous Energy-Conservation Neural Network for Structural Dynamics Analysis

Yuan

Wang

Yang

et al. 2020

Preprint

View full text Add to dashboard Cite

Fast and accurate structural dynamics analysis is important for structural design and damage assessment. Structural dynamics analysis leveraging machine learning techniques has become a popular research focus in recent years. Although the basic neural network provides an alternative approach for structural dynamics analysis, the lack of physics law inside the neural network limits the model accuracy and fidelity. In this paper, a new family of the energy-conservation neural network is introduced, which respects the physical laws. The neural network is explored from a fundamental single-degree-of-freedom system to a complicated multiple-degrees-of-freedom system. The damping force and external forces are also considered step by step. To improve the parallelization of the algorithm, the derivatives of the structural states are parameterized with the novel energy-conservation neural network instead of specifying the discrete sequence of structural states. The proposed model uses the system energy as the last layer of the neural network and leverages the underlying automatic differentiation graph to incorporate the system energy naturally, which ultimately improves the accuracy and long-term stability of structures dynamics response calculation under an earthquake impact. The trade-off between computation accuracy and speed is discussed. As a case study, a 3-story building earthquake simulation is conducted with realistic earthquake records.

show abstract

Convolutional Neural Networks combined with Runge-Kutta Methods

Cited by 15 publications

References 20 publications

LT-OCF: Learnable-Time ODE-based Collaborative Filtering

LT-OCF: Learnable-Time ODE-based Collaborative Filtering

TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction

Time-Continuous Energy-Conservation Neural Network for Structural Dynamics Analysis

Contact Info

Product

Resources

About