SlimNets: An Exploration of Deep Model Compression and Acceleration

Abstract

Deep neural networks have achieved increasingly accurate results on a wide variety of complex tasks. However, much of this improvement is due to the growing use and availability of computational resources (e.g use of GPUs, more layers, more parameters, etc). Most state-of-the-art deep networks, despite performing well, over-parameterize approximate functions and take a significant amount of time to train. With increased focus on deploying deep neural networks on resource constrained devices like smart phones, there has been a push to evaluate why these models are so resource hungry and how they can be made more efficient. This work evaluates and compares three distinct methods for deep model compression and acceleration: weight pruning, low rank factorization, and knowledge distillation. Comparisons on VGG nets trained on CIFAR10 show that each of the models on their own are effective, but that the true power lies in combining them. We show that by combining pruning and knowledge distillation methods we can create a compressed network 85 times smaller than the original, all while retaining 96% of the original model's accuracy.

Deep Learning Methods for Solving Linear Inverse Problems: Research Directions and Paradigms

“…Representative work include the SqueezeNet [243] and the MobileNet [240]. Another method is compressing an existing network to decrease the number of parameters and the required computation resource, under the guarantee of reconstruction accuracy [244,245,246,247,248]. For example, the model cutting method compresses the model by cutting unimportant connections of a trained model according to some effective evaluations [249].…”

AntMan: Sparse Low-Rank Compression to Accelerate RNN inference

“…Teacher-Student training paradigm: Knowledge Distillation (KD) developed by Hinton et al (2015) is a popular technique to compress deep and wide networks into sparser ones, where the compressed model mimics the distribution learned by the complex model. Oguntola et al (2018) show that compression techniques such as pruning, and low-rank decomposition can be combined with KD to significantly improve compression rate, while maintaining accuracy. KD usually optimizes a weighted average of two different objective functions.…”

“…Pruning+KD s combines pruning with knowledge distillation using KL-divergence as described in SlimNets (Oguntola et al, 2018). Pruning+KD o applies pruning together with our version of knowledge distillation (KD o ) that combines KL-divergence with MSE-loss.…”

Learning Frequency Domain Approximation for Binary Neural Networks

“…The success of deep convolutional neural networks (CNNs) has been well demonstrated in several real-world applications, e.g. , image classification [20,29], object detection [35], semantic segmentation [28], and low-level computer vision [37]. Massive parameters and huge computational complexity are usually required for achieving the desired high performance, which limits the application of these models to portable devices such as mobile phones and smart cameras.…”