Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?

Abstract

Binary neural networks (BNN) have been studied extensively since they run dramatically faster at lower memory and power consumption than floating-point networks, thanks to the efficiency of bit operations. However, contemporary BNNs whose weights and activations are both single bits suffer from severe accuracy degradation. To understand why, we investigate the representation ability, speed and bias/variance of BNNs through extensive experiments. We conclude that the error of BNNs are predominantly caused by the intrinsic instability (training time) and non-robustness (train & test time). Inspired by this investigation, we propose the Binary Ensemble Neural Network (BENN) which leverages ensemble methods to improve the performance of BNNs with limited efficiency cost. While ensemble techniques have been broadly believed to be only marginally helpful for strong classifiers such as deep neural networks, our analysis and experiments show that they are naturally a perfect fit to boost BNNs. We find that our BENN, which is faster and more robust than state-of-the-art binary networks, can even surpass the accuracy of the full-precision floating number network with the same architecture.

Binary neural networks: A survey

“…cial characteristics of BNNs are necessary. In fact, some of the recent works essentially worked on this such as XNOR-Net++[76], CBCN[88], Self-Binarizing Networks[94], BENN[67], etc. The results show that specially designed methods considering characteristics of BNNs can achieve better performance.…”

TentacleNet: A Pseudo-Ensemble Template for Accurate Binary Convolutional Neural Networks

“…For all the three benchmarks, additional experiments revealed the accuracy of TentacleNet saturates, namely, there is no further improvement by increasing the number of tentacles; the top right points of the three lines in the plot of Fig. 3 a quantitative comparison against BENN [16], which is state-of-the-art for binary ensembles. The BENN strategy is to apply standard ensemble methods to BNNs, bagging and boosting in particular.…”

“…Some recent works explored this option. For instance, in [16] the authors adapted the classical ensemble methods to binary CNNs, bagging and boosting in particular. The collected results revealed that a large number of BNNs is needed to get close to the accuracy of the full-precision model, thus resulting in large memory space.…”

“…Experimental results collected over three computer vision tasks, i.e. image classification on CIFAR-10 and CIFAR-100 [14] and facial expression recognition on the FER13 data-set [15], reveal TentacleNet can reach the accuracy of full-precision models, yet ensuring much lower memory footprint compared to state-of-the-art binary ensemble methods [16]. In [5], the same authors presented a full-binary CNN where also the activations get projected in a binary space using a sign activation function.…”

Deep Neural Network Approximation for Custom Hardware

“…For the CIFAR-10 dataset, for example, binary networks such as FINN and ReBNet require a wider and deeper model, CNV, in order to achieve similar accuracy to an FP32 baseline with CifarNet, a much thinner and shallower model [130]. Zhu et al proposed the Binary Ensemble Neural Network (BENN), in which multiple binarised networks are aggregated by "boosting" (parallel ensemble with trained weights) [163]. e authors showed that their network ensembles exhibited lower bias and variance than their individual constituents while also having improved robustness to noise.…”