Self-distribution binary neural networks

Xue, Ping; Chang, Jingfei; Wei, Xing; Wei, Zhen

doi:10.1007/s10489-022-03348-z

Cited by 10 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another binary network based on MobileNetV1 is ReActNet [87] which proposed ReAct-Sign (RSign) and ReAct-PReLU (RPReLU) as alternatives of the traditional activation function to reshape the activation distribution. While in [88], the authors proposed Activation Self Distribution (ASD) and Weight Self Distribution (WSD) to adjust the sign distribution of activations and weights, respectively, to enhance the accuracy.…”

Section: C: Gradient Error Minimizationmentioning

confidence: 99%

A Systematic Literature Review on Binary Neural Networks

et al. 2023

View full text Add to dashboard Cite

This paper presents an extensive literature review on Binary Neural Network (BNN). BNN utilizes binary weights and activation function parameters to substitute the full-precision values. In digital implementations, BNN replaces the complex calculations of Convolutional Neural Networks (CNNs) with simple bitwise operations. BNN optimizes large computation and memory storage requirements, which leads to less area and power consumption compared to full-precision models. Although there are many advantages of BNN, the binarization process has a significant impact on the performance and accuracy of the generated models. To reflect the state-of-the-art in BNN and explore how to develop and improve BNNbased models, we conduct a systematic literature review on BNN with data extracted from 239 research studies. Our review discusses various BNN architectures and the optimization approaches developed to improve their performance. There are three main research directions in BNN: accuracy optimization, compression optimization, and acceleration optimization. The accuracy optimization approaches include quantization error reduction, special regularization, gradient error minimization, and network structure. The compression optimization approaches combine fractional BNN and pruning. The acceleration optimization approaches comprise computing in-memory, FPGA-based implementations, and ASIC-based implementations. At the end of our review, we present a comprehensive analysis of BNN applications and their evaluation metrics. Also, we shed some light on the most common BNN challenges and the future research trends of BNN.

show abstract

Section: C: Gradient Error Minimizationmentioning

confidence: 99%

A Systematic Literature Review on Binary Neural Networks

et al. 2023

View full text Add to dashboard Cite

show abstract

“…We first evaluate our method on CIFAR10 dataset by comparing it with existing state-of-the-arts binary quantization methods, including DSQ [7], DoReFa [22], IR-Net [16], L2B [21] and SD-BNN [20]. Among them, DSQ [7] and IR-Net [16] propose to use T anh as soft functions to approximate the Sign function.…”

Section: Cifar10mentioning

confidence: 99%

“…We will further verify this view in section 5.3. 1/1 85.3 IR-Net [16] 1/1 86.5 SD-BNN [20] 1/1 86.9 Ours 1/1 87.9…”

Section: Cifar10mentioning

confidence: 99%

Binarizing by Classification: Is soft function really necessary?

He¹,

Zhang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Binary neural network leverages the Sign function to binarize real values, and its non-derivative property inevitably brings huge gradient errors during backpropagation. Although many hand-designed soft functions have been proposed to approximate gradients, their mechanism is not clear and there are still huge performance gaps between binary models and their full-precision counterparts. To address this, we propose to tackle network binarization as a binary classification problem and use a multi-layer perceptron (MLP) as the classifier. The MLP-based classifier can fit any continuous function theoretically and is adaptively learned to binarize networks and backpropagate gradients without any specific soft function. With this view, we further prove experimentally that even a simple linear function can outperform previous complex soft functions. Extensive experiments demonstrate that the proposed method yields surprising performance both in image classification and human pose estimation tasks. Specifically, we achieve 65.7% top-1 accuracy of ResNet-34 on ImageNet dataset, with an absolute improvement of 2.8%. When evaluating on the challenging Microsoft COCO keypoint dataset, the proposed method enables binary networks to achieve a mAP of 60.6 for the first time, on par with some full-precision methods.Preprint. Under review.

show abstract

“…This implies the new weight distribution has an equal amount of weights below and above zero, and therefore after binarization, the amount of -1's and +1's is equal and entropy is maximized. SD-BNN (2021) [50] employs a series of linear layers and non-linearities to calculate biases for both weights and features. Note that this approach for the features belongs in the category of network topology changing.…”

Section: Normalizationmentioning

confidence: 99%

“…R20C10 experiments will use default data augmentation and train for 500 epochs, whereas R18C100 experiments will use AutoAug data augmentation and train for 1000 epochs. [46] 91.10% Circulant BNN [28] 69.97% IR-Net [34] 86.5% Real-to-Binary [32] 76.2% BBG [39] 85.34% ProxyBNN [17] 67.17% RBNN [25] 87.8% Information capacity BNN [19] 73.48% Noisy Supervision [14] 85.78% ReCU [49] 69.1% ReCU [49] 87.4% BNN-BN [8] 68.34% Sub-bit BNN [45] 83.9% Equal Bits [24] 71.60% SD-BNN [50] 86.9% BNN fully latent weights [48] 88.6% Fig. 16 Baseline BNN compared to different feature binarizers.…”

Section: Design Space Explorationmentioning

confidence: 99%

How to train accurate BNNs for embedded systems?

Putter¹,

Corporaal²

2022

Preprint

View full text Add to dashboard Cite

A key enabler of deploying convolutional neural networks on resourceconstrained embedded systems is the binary neural network (BNN). BNNs save on memory and simplify computation by binarizing both features and weights. Unfortunately, binarization is inevitably accompanied by a severe decrease in accuracy. To reduce the accuracy gap between binary and full-precision networks, many repair methods have been proposed in the recent past, which we have classified and put into a single overview in this chapter. The repair methods are divided into two main branches, training techniques and network topology changes, which can further be split into smaller categories. The latter category introduces additional cost (energy consumption or additional area) for an embedded system, while the former does not. From our overview, we observe that progress has been made in reducing the accuracy gap, but BNN papers are not aligned on what repair methods should be used to get highly accurate BNNs. Therefore, this chapter contains an empirical review that evaluates the benefits of many repair methods in isolation over the ResNet-20&CIFAR10 and ResNet-18&CIFAR100 benchmarks. We found three repair categories most beneficial: feature binarizer, feature normalization, and double residual. Based on this review we discuss future directions and research opportunities. We sketch the benefit and costs associated with BNNs on embedded systems because it remains to be seen whether BNNs will be able to close the accuracy gap while staying highly energy-efficient on resource-constrained embedded systems.

show abstract

Self-distribution binary neural networks

Cited by 10 publications

References 36 publications

A Systematic Literature Review on Binary Neural Networks

A Systematic Literature Review on Binary Neural Networks

Binarizing by Classification: Is soft function really necessary?

How to train accurate BNNs for embedded systems?

Contact Info

Product

Resources

About