Compressing Convolutional Neural Networks in the Frequency Domain

Abstract

Convolutional neural networks (CNN) are increasingly used in many areas of computer vision. They are particularly attractive because of their ability to "absorb" great quantities of labeled data through millions of parameters. However, as model sizes increase, so do the storage and memory requirements of the classifiers, hindering many applications such as image and speech recognition on mobile phones and other devices. In this paper, we present a novel network architecture, Frequency-Sensitive Hashed Nets (FreshNets), which exploits inherent redundancy in both convolutional layers and fully-connected layers of a deep learning model, leading to dramatic savings in memory and storage consumption. Based on the key observation that the weights of learned convolutional filters are typically smooth and low-frequency, we first convert filter weights to the frequency domain with a discrete cosine transform (DCT) and use a low-cost hash function to randomly group frequency parameters into hash buckets. All parameters assigned the same hash bucket share a single value learned with standard backpropagation. To further reduce model size, we allocate fewer hash buckets to high-frequency components, which are generally less important. We evaluate FreshNets on eight data sets, and show that it leads to better compressed performance than several relevant baselines.

Opportunities and obstacles for deep learning in biology and medicine

“…Much has been done to minimize the memory requirements of neural networks [445,[493][494][495][496]506,507], but there is also growing interest in specialized hardware, such as field-programmable gate arrays (FPGAs) [502,508] and application-specific integrated circuits (ASICs) [509]. Less software is available for such highly specialized hardware [508].…”

Improved Techniques for Training Adaptive Deep Networks

“…One stream focuses on designing efficient network architectures [30,28,15,40,13], including depthwise separable convolution [30], point-wise group convolution with channel shuffling [39], and learned group convolution [15], to name a few. The other line of research explores methods to prune [8,23,11] or quantize [4,8,17] neural network weights. These strategies are effective when neural networks have a substantial amount of redundant weights, which can be safely removed or quantized without sacrificing accuracy.…”

“…Extensive efforts have been made to improve the inference efficiency of deep CNNs in recent years. Popular approaches include efficient architecture design [30,28,15,40], network pruning [8,23,26], weight quantiza- * First two authors contributed equally † Corresponding author tion [4,8,17] and adaptive inference [7,14,2,35,6,34]. Among them, adaptive inference is gaining increasing attention recently, due to its remarkable advantages.…”

Deep compression of convolutional neural networks with low-rank approximation

“…Yoon and Hwang enforce sparsity on the filter through regularization. Chen and others also explored the frequency domain for compression . However, all of these methods are more complex when implemented using standard deep‐learning tools, which was actually the reason why we did not choose them as candidates for our approach.…”