Network Sketching: Exploiting Binary Structure in Deep CNNs

Abstract

Convolutional neural networks (CNNs) with deep architectures have substantially advanced the state-of-the-art in computer vision tasks. However, deep networks are typically resource-intensive and thus difficult to be deployed on mobile devices. Recently, CNNs with binary weights have shown compelling efficiency to the community, whereas the accuracy of such models is usually unsatisfactory in practice. In this paper, we introduce network sketching as a novel technique of pursuing binary-weight CNNs, targeting at more faithful inference and better trade-off for practical applications. Our basic idea is to exploit binary structure directly in pre-trained filter banks and produce binaryweight models via tensor expansion. The whole process can be treated as a coarse-to-fine model approximation, akin to the pencil drawing steps of outlining and shading. To further speedup the generated models, namely the sketches, we also propose an associative implementation of binary tensor convolutions. Experimental results demonstrate that a proper sketch of AlexNet (or ResNet) outperforms the existing binary-weight models by large margins on the ImageNet large scale classification task, while the committed memory for network parameters only exceeds a little.

Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation

“…BNNs [23,40] propose to constrain both weights and activations to binary values (i.e., +1 and -1), where the multiply-accumulations can be replaced by purely xnor(·) and popcount(·) operations. To make a trade-off between accuracy and complexity, [13,15,29,48] propose to recursively perform residual quantization and yield a series of binary tensors with decreasing magnitude scales. However, multiple binarizations are sequential process which cannot be paralleled.…”

“…We explore the difference between layer-wise and group-wise design strategies in approach can be treated as a kind of tensor approximation which has similarities with multiple binarizations methods in [13,15,29,30,48] and the differences are described in Sec. 4.…”

Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?

“…They have demonstrated the power of BNNs in terms of speed, memory use and power consumption. But recent works such as [58,11,21,10] also reveal the strong accuracy degradation and mismatch issue during the training when BNNs are applied in complicated tasks such as ImageNet ( [12]) recognition, especially when the activation is binarized. Although some work like [43,50,13] have offered reasonable solutions to approximate fullprecision neural network, much more computation and tricks on hyperparameters are still needed to implement compared with BENN.…”

“…Compute activation a l based on binary kernel w l b and input a l−1 ; 14 end 15 Backward Pass: 16 Compute gradient ∂J ∂wt based on [50,28]; 17 Parameter Update: 18 Update w t to w t+1 with any update rules (e.g., SGD or ADAM) 19 end 20 Ensemble Update: 21 Pick the BNN when training converges; 22 Use either bagging or boosting algorithm to update weight u i of each training example i; 23 end 24 Return: K trained base classifiers for BENN;…”

DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition

“…Wen et al [40] proposes a method as shown in (7), where s t := g t ∞ := max(abs(g t )) is a scaler parameter, ⊗ is the Hadamard product, abs(·) respectively returns the absolute value of each element. The method quantizes gradients to ternary values that can effectively improve clients-to-server communication in distributed learning.g Guo et al [41] propose greedy approximation, which instead tries to learn the quantization as shown in (8), where B i is a binary filter, α i are optimization parameters and input channels (c)× width (w)× height (h) is the size of the filter.…”