Energy Propagation in Deep Convolutional Neural Networks

Abstract

Many practical machine learning tasks employ very deep convolutional neural networks. Such large depths pose formidable computational challenges in training and operating the network. It is therefore important to understand how fast the energy contained in the propagated signals (a.k.a. feature maps) decays across layers. In addition, it is desirable that the feature extractor generated by the network be informative in the sense of the only signal mapping to the all-zeros feature vector being the zero input signal. This "trivial null-set" property can be accomplished by asking for "energy conservation" in the sense of the energy in the feature vector being proportional to that of the corresponding input signal. This paper establishes conditions for energy conservation (and thus for a trivial null-set) for a wide class of deep convolutional neural network-based feature extractors and characterizes corresponding feature map energy decay rates. Specifically, we consider general scattering networks employing the modulus non-linearity and we find that under mild analyticity and high-pass conditions on the filters (which encompass, inter alia, various constructions of Weyl-Heisenberg filters, wavelets, ridgelets, (α)-curvelets, and shearlets) the feature map energy decays at least polynomially fast. For broad families of wavelets and Weyl-Heisenberg filters, the guaranteed decay rate is shown to be exponential. Moreover, we provide handy estimates of the number of layers needed to have at least ((1 − ε) · 100)% of the input signal energy be contained in the feature vector.Index Terms-Machine learning, deep convolutional neural networks, scattering networks, energy decay and conservation, frame theory.

Graph convolutional neural networks via scattering

“…The first equality in (20) clearly follows from (30) and (31). The second equality in (20) is an immediate consequence of the first equality and the observation that for…”

“…Numerical results illustrating the energy decay in the Euclidean domain are given in [19]. Furthermore, theoretical rates are provided in [20] and [21], where [20] introduces additional assumptions on the smoothness of input signals and the bandwidth of filters and [21] studies time-frequency frames instead of wavelets.…”

Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks

“…see [5]). More recently, interesting results have focused on the expressive ability of deeper and sparser architectures [48,9,27,52]. Computational tractability of training networks however is still a major challenge.…”

Basic filters for convolutional neural networks applied to music: Training or design?

“…Remark 3.1. It has been observed in [19,22,24,25] that in the context of scattering networks, most of the input signal's energy is contained in the output of the first two convolutional layers. While the context and the filters here are different, this observation might be interesting also as a background for the usual choice of architecture of CNNs for audio processing.…”