HadaNets: Flexible Quantization Strategies for Neural Networks

Abstract

On-board processing elements on UAVs are currently inadequate for training and inference of Deep Neural Networks. This is largely due to the energy consumption of memory accesses in such a network. HadaNets introduce a flexible train-from-scratch tensor quantization scheme by pairing a full precision tensor to a binary tensor in the form of a Hadamard product. Unlike wider reduced precision neural network models, we preserve the train-time parameter count, thus out-performing XNOR-Nets without a traintime memory penalty. Such training routines could see great utility in semi-supervised online learning tasks. Our method also offers advantages in model compression, as we reduce the model size of ResNet-18 by 7.43× with respect to a full precision model without utilizing any other compression techniques. We also demonstrate a 'Hadamard Binary Matrix Multiply' kernel, which delivers a 10-fold increase in performance over full precision matrix multiplication with a similarly optimized kernel.

PokeBNN: A Binary Pursuit of Lightweight Accuracy

“…Many of them aim to incur negligible compute and parameter overhead. For example, a channelwise realvalued rescaling of the binarized tensors can effectively mitigate the quantization loss [2,7,42]. Connecting the unquantized input activations of a binarized convolutional layer to its output with a shortcut enhances the gradient flow and the model representation capacity [35].…”