PokeBNN: A Binary Pursuit of Lightweight Accuracy

Abstract

Top-1 ImageNet optimization promotes enormous networks that may be impractical in inference settings. Binary neural networks (BNNs) have the potential to significantly lower the compute intensity but existing models suffer from low quality. To overcome this deficiency, we propose Poke-Conv, a binary convolution block which improves quality of BNNs by techniques such as adding multiple residual paths, and tuning the activation function. We apply it to ResNet-50 and optimize ResNet's initial convolutional layer which is hard to binarize. We name the resulting network family PokeBNN 1 . These techniques are chosen to yield favorable improvements in both top-1 accuracy and the network's cost. In order to enable joint optimization of the cost together with accuracy, we define arithmetic computation effort (ACE), a hardware-and energy-inspired cost metric for quantized and binarized networks. We also identify a need to optimize an under-explored hyper-parameter controlling the binarization gradient approximation.We establish a new, strong state-of-the-art (SOTA) on top-1 accuracy together with commonly-used CPU64 cost, ACE cost and network size metrics. , the previous SOTA in BNNs, achieved a 70.5% top-1 accuracy with 7.9 ACE. A small variant of PokeBNN achieves 70.5% top-1 with 2.6 ACE, more than 3x reduction in cost; a larger PokeBNN achieves 75.6% top-1 with 7.8 ACE, more than 5% improvement in accuracy without increasing the cost. PokeBNN implementation in JAX/Flax [6,18] and reproduction instructions are open sourced. 2 * Work performed while at Google, equal contribution. 1 Poke/'p6kI/ is pronounced similarly to pocket. PokeConv, PokeBNN, and Pokemon are abbreviations of Pocket Convolution, Pocket Binary Neural Network, and Pocket Monster, respectively.2 Source code is available as part of the supplemental materials. The camera-ready version will include the GitHub link and optimize the source for clarity and ease of reproduction.

LAB: Learnable Activation Binarizer for Binary Neural Networks

“…The research in BNNs is focussed on bringing deep learning to resourceconstrained edge devices. Recent studies report the computational complexity of their models using theoretical metrics such as floating-point operations (FLOPs) [24,27] multiplyaccumulate (MACs) [4] or arithmetic computation effort (ACE) [40]. In coherence with [3,30] we argue that latency is the best metric to compare model performances.…”

INSTA-BNN: Binary Neural Network with INSTAnce-aware Threshold

“…Note that while the compute cost of a SE-like module is usually considered to be negligible, its parameter size cannot be ignored in BNN models. We calculated total number of opertations (OPs) as OPs = FLOPs + (BOPs / 64) + (int4 OPs / 16), following [18,19,32]. In case of parameters, binary weights are 1-bit, weights of SE-like modules are 8-bit, and other real-valued parameters and weights are considered as 32-bit.…”

Empirical Evaluation of Post-Training Quantization Methods for Language Tasks

“…We use Arithmetic Computation Effort (ACE) (Zhang et al, 2021), a newly proposed hardware-and energy-inspired cost metric to evaluate the inference cost of quantized BERT models. ACE is defined as Table 1: We quantize 32-bit baseline models to 8-bits by three quantization methods.…”

“…I and J are sets of all quantization bits used for inference. ACE is shown to be well correlated to the actual energy consumption on Google TPUs hardware and used to evaluate the inference cost of Binary Neural Networks in Zhang et al (2021).…”