Stable architectures for deep neural networks

Abstract

Deep neural networks have become invaluable tools for supervised machine learning, e.g., classification of text or images. While often offering superior results over traditional techniques and successfully expressing complicated patterns in data, deep architectures are known to be challenging to design and train such that they generalize well to new data. Critical issues with deep architectures are numerical instabilities in derivative-based learning algorithms commonly called exploding or vanishing gradients. In this paper, we propose new forward propagation techniques inspired by systems of Ordinary Differential Equations (ODE) that overcome this challenge and lead to well-posed learning problems for arbitrarily deep networks.The backbone of our approach is our interpretation of deep learning as a parameter estimation problem of nonlinear dynamical systems. Given this formulation, we analyze stability and well-posedness of deep learning and use this new understanding to develop new network architectures. We relate the exploding and vanishing gradient phenomenon to the stability of the discrete ODE and present several strategies for stabilizing deep learning for very deep networks. While our new architectures restrict the solution space, several numerical experiments show their competitiveness with state-of-the-art networks.

Variational Networks: An Optimal Control Approach to Early Stopping Variational Methods for Image Restoration

“…For image deblurring, images computed with too small values of T remain blurry, while for T > T ringing artifacts are generated and their intensity increase with larger T . For a corrupted image, the associated adjoint state requires the knowledge of the ground truth for the terminal condition (14), which is in general not available. However, Figure 7 shows that the learned average optimal stopping time T yields the smallest expected error.…”

A deep learning enabler for nonintrusive reduced order modeling of fluid flows

“…Deep neural networks are capable of approximating nonlinear dynamical systems as shown in many studies 102,103,106,123 . The general nonlinear dynamical system can be presented by an equation of the form…”

Learning Adaptive Regularization for Image Labeling Using Geometric Assignment

“…The authors adopted the discretize-then-differentiate viewpoint on the parameter estimation problem and suggested symplectic numerical integration in order to achieve better stability. As mentioned above, our work contrasts in that inference is always exact during learning, unlike the more involved architecture of [HR17] where learning is based on approximate inference. Furthermore, in our case, symplectic numerical integration is a consequence of making the diagram of Figure 2.2 (page 8) commute.…”