A Main/Subsidiary Network Framework for Simplifying Binary Neural Networks

Xu, Yinghao; Dong, Xin; Li, Yudian; Su, Hao

doi:10.1109/cvpr.2019.00732

Cited by 26 publications

(15 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The process of knowledge distillation is shown in Figure 3. Similar mimic solutions like Distillation and Quantization (DQ) [81], Distilled Binary Neural Network (DBNN) [80] and Main/Subsidiary Network [87] have been studied, and their experiments demonstrate that the loss functions related to the full-precision teacher model help to stabilize the training of binary student model with high accuracy. CI-BCNN proposed in [86] mines the channel-wise interactions, through which prior knowledge is provided to alleviate inconsistency of signs in binary feature maps and preserves the information of input samples during inference.…”

Section: Improve the Network Loss Functionmentioning

confidence: 99%

Binary neural networks: A survey

Qin

Gong

Liu

et al. 2020

Pattern Recognition

382

171

View full text Add to dashboard Cite

The binary neural network, largely saving the storage and computation, serves as a promising technique for deploying deep models on resource-limited devices.However, the binarization inevitably causes severe information loss, and even worse, its discontinuity brings difficulty to the optimization of the deep network.To address these issues, a variety of algorithms have been proposed, and achieved satisfying progress in recent years. In this paper, we present a comprehensive survey of these algorithms, mainly categorized into the native solutions directly conducting binarization, and the optimized ones using techniques like minimizing the quantization error, improving the network loss function, and reducing the gradient error. We also investigate other practical aspects of binary neural networks such as the hardware-friendly design and the training tricks. Then, we give the evaluation and discussions on different tasks, including image classification, object detection and semantic segmentation. Finally, the challenges that may be faced in future research are prospected.the heavy computation and storage still inevitably limit the applications of the deep CNNs in practice. Besides, due to the huge model parameter space, the prediction of the neural networks is usually viewed as a black-box, which brings great challenges to the interpretability of CNNs. Some works like [21,22,23] empirically explore the function of each layer in the network. They visualize the feature maps extracted by different filters and view each filter as a visual unit focusing on different visual components.of the ResNet-50 [28], and meanwhile save more than 75% of parameters and 50% computational time. In the literature, approaches for compressing the deep networks can be classified into five categories: parameter pruning [26,29,30,31], parameter quantizing [32,33,34,35,36,37,38,39,40,41], low-rank parameter factorization [42,43,44,45,46], transferred/compact convolutional filters [47,48,49,50], and knowledge distillation [51,52,53,54,55,56]. The parameter pruning and quantizing mainly focus on eliminating the redundancy in the model parameters respectively by removing the redundant/uncritical ones or compressing the parameter space (e.g. , from the floating-point weights to the integer ones). Low-rank factorization applies the matrix/tensor decomposition techniques to estimate the informative parameters using the proxy ones of small size. The compact convolutional filter based approaches rely on the carefullydesigned structural convolutional filters to reduce the storage and computation complexity. The knowledge distillation methods try to distill a more compact model to reproduce the output of a larger network.Among the existing network compression techniques, quantization based one serves as a promising and fast solution that yields highly compact models compared to their floating-point counterparts, by representing the network weights with very low precision. Along this direction, the most extreme quantization is binarization, the interest...

show abstract

Section: Improve the Network Loss Functionmentioning

confidence: 99%

Binary neural networks: A survey

Qin

Gong

Liu

et al. 2020

Pattern Recognition

382

171

View full text Add to dashboard Cite

show abstract

“…BinaryConnect [17] 1/32 VGG-Small 91.7 BNN [18] 1/1 VGG-Small 89.9 XNOR-Net [19] 1/1 VGG-Small 89.8 LQ-Nets [20] 1/32 ResNet-20 90.1 BBG [21] 1/1 ResNet-20 85.3 BCGD [22] 1/4 VGG-11 89.6 IR-Net [23] 1/1 VGG-Small 90.4 CI-BCNN [24] 1/1 VGG-Small 92.5 Multi-scale BNN(Ours)…”

Section: Methodsmentioning

confidence: 99%

Implementation of Binarized Neural Networks in All-Programmable System-on-Chip Platforms

Xiang¹,

Teo²

2022

Electronics

View full text Add to dashboard Cite

The Binarized Neural Network (BNN) is a Convolutional Neural Network (CNN) consisting of binary weights and activation rather than real-value weights. Smaller models are used, allowing for inference effectively on mobile or embedded devices with limited power and computing capabilities. Nevertheless, binarization results in lower-entropy feature maps and gradient vanishing, which leads to a loss in accuracy compared to real-value networks. Previous research has addressed these issues with various approaches. However, those approaches significantly increase the algorithm’s time and space complexity, which puts a heavy burden on those embedded devices. Therefore, a novel approach for BNN implementation on embedded systems with multi-scale BNN topology is proposed in this paper, from two optimization perspectives: hardware structure and BNN topology, that retains more low-level features throughout the feed-forward process with few operations. Experiments on the CIFAR-10 dataset indicate that the proposed method outperforms a number of current BNN designs in terms of efficiency and accuracy. Additionally, the proposed BNN was implemented on the All Programmable System on Chip (APSoC) with 4.4 W power consumption using the hardware accelerator.

show abstract

“…6, the binarized CompConv beat binarized Ghost both in parameters and FLOPs, with nearly 2× acceleration and storage saving. Though the binarized model is lightweight enough, there still exists redundancy of computation cost as pointed by [38], the computation efficiency of which can also be improved by replacing regular convolution with our CompConv. We also apply CompConv on ResNet to verify its capability of finer classification on CIFAR-100.…”

Section: Binarized Compconv On Cifar-10mentioning

confidence: 99%

CompConv: A Compact Convolution Module for Efficient Feature Learning

Zhang¹,

Xu²,

Shen³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Convolutional Neural Networks (CNNs) have achieved remarkable success in various computer vision tasks but rely on tremendous computational cost. To solve this problem, existing approaches either compress well-trained large-scale models or learn lightweight models with carefully designed network structures. In this work, we make a close study of the convolution operator, which is the basic unit used in CNNs, to reduce its computing load.In particular, we propose a compact convolution module, called CompConv, to facilitate efficient feature learning. With the divide-and-conquer strategy, CompConv is able to save a great many computations as well as parameters to produce a certain dimensional feature map. Furthermore, CompConv discreetly integrates the input features into the outputs to efficiently inherit the input information. More importantly, the novel CompConv is a plug-andplay module that can be directly applied to modern CNN structures to replace the vanilla convolution layers without further effort. Extensive experimental results suggest that CompConv can adequately compress the benchmark CNN structures yet barely sacrifice the performance, surpassing other competitors.

show abstract

A Main/Subsidiary Network Framework for Simplifying Binary Neural Networks

Cited by 26 publications

References 18 publications

Binary neural networks: A survey

Binary neural networks: A survey

Implementation of Binarized Neural Networks in All-Programmable System-on-Chip Platforms

CompConv: A Compact Convolution Module for Efficient Feature Learning

Contact Info

Product

Resources

About