Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution

Abstract

In natural images, information is conveyed at different frequencies where higher frequencies are usually encoded with fine details and lower frequencies are usually encoded with global structures. Similarly, the output feature maps of a convolution layer can also be seen as a mixture of information at different frequencies. In this work, we propose to factorize the mixed feature maps by their frequencies, and design a novel Octave Convolution (OctConv) operation 1 to store and process feature maps that vary spatially "slower" at a lower spatial resolution reducing both memory and computation cost. Unlike existing multi-scale methods, OctConv is formulated as a single, generic, plug-andplay convolutional unit that can be used as a direct replacement of (vanilla) convolutions without any adjustments in the network architecture. It is also orthogonal and complementary to methods that suggest better topologies or reduce channel-wise redundancy like group or depth-wise convolutions. We experimentally show that by simply replacing convolutions with OctConv, we can consistently boost accuracy for both image and video recognition tasks, while reducing memory and computational cost. An OctConv-equipped ResNet-152 can achieve 82.9% top-1 classification accuracy on ImageNet with merely 22.2 GFLOPs.

Res2Net: A New Multi-Scale Backbone Architecture

“…Recently, there are some concurrent works aiming at improving the performance by utilizing the multi-scale features [5], [9], [11], [49]. Big-Little Net [5] is a multi-branch network composed of branches with different computational complexity.…”

“…Big-Little Net [5] is a multi-branch network composed of branches with different computational complexity. Octave Conv [9] decomposes the standard convolution into two resolutions to process features at different frequencies. MSNet [11] utilizes a high-resolution network to learn high-frequency residuals by using the up-sampled low-resolution features learned by a lowresolution network.…”

“…Unlike most existing methods that enhance the layer-wise multi-scale representation strength of CNNs, we improve the multi-scale representation ability at a more granular level. Different from some concurrent works [5], [9], [11] that improve the multi-scale ability by utilizing features with different resolutions, the multi-scale of our proposed method refers to the multiple available receptive fields at a more granular level. To achieve this goal, we replace the 3 × 3 filters 1 of n channels, with a set of smaller filter groups, each with w channels (without loss of generality we use n = s × w).…”

Multi-Scale Octave Convolutions for Robust Speech Recognition

“…An octave convolutional layer [1] factorizes the output feature maps of a convolutional layer into two groups. The resolution of the low-frequency feature maps is reduced by an octaveheight and width dimensions are divided by 2.…”

“…

We propose a multi-scale octave convolution layer to learn robust speech representations efficiently. Octave convolutions were introduced by Chen et al [1] in the computer vision field to reduce the spatial redundancy of the feature maps by decomposing the output of a convolutional layer into feature maps at two different spatial resolutions, one octave apart. This approach improved the efficiency as well as the accuracy of the CNN models.

…”