Compact and Computationally Efficient Representation of Deep Neural Networks

Abstract

At the core of any inference procedure in deep neural networks are dot product operations, which are the component that require the highest computational resources. For instance, deep neural networks such as VGG-16 require up to 15 gigaoperations in order to perform the dot products present in a single forward pass, which results in significant energy consumption and therefore limit their use in resource-limited environments, e.g., on embedded devices or smartphones. A common approach to reduce the cost of inference is to reduce its memory complexity by lowering the entropy of the weight matrices of the neural network, e.g., by pruning and quantizing their elements. However, the quantized weight matrices are then usually represented either by a dense or sparse matrix storage format, whose associated dot product complexity is not bounded by the entropy of the matrix. This means that the associated inference complexity ultimately depends on the implicit statistical assumptions that these matrix representations make about the weight distribution, which can be in many cases suboptimal. In this paper we address this issue and present new efficient representations for matrices with low entropy statistics. These new matrix formats have the novel property that their memory and algorithmic complexity are implicitly bounded by the entropy of the matrix, consequently implying that they are guaranteed to become more efficient as the entropy of the matrix is being reduced. In our experiments we show that performing the dot product under these new matrix formats can indeed be more energy and time efficient under practically relevant assumptions. For instance, we are able to attain up to x42 compression ratios, x5 speed ups and x90 energy savings when we convert in a lossless manner the weight matrices of state-of-the-art networks such as AlexNet, VGG-16, ResNet152 and DenseNet into the new matrix formats and benchmark their respective dot product operation.

DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks

“…These matrix data structures do not only offer compression gains, but also an efficient execution of the associated dot product algorithm [54]. Similarly, [14] proposed two novel matrix representation, the Compressed Entropy Row (CER) and Compressed Shared Elements Row (CSER) representations, that are provably more optimal than the CSR with regards to both, compression and execution efficiency when the networks parameters have low entropy statistics.…”

Multi-Kernel Prediction Networks for Denoising of Burst Images

“…The computational cost of the in-place additions is negligible. Note that model compression [23,24] and efficient representations [25] can further reduce the computational costs. Fig.…”

“…(a) input burst (b) KPN[14] (c) KPN L25 (d) MKPN (e) ground truth Example of denoising an image of a bear at Gain ∝ 4. The detailed fur is recovered best by MKPN.…”

Digital multiplier‐less implementation of high‐precision SDSP and synaptic strength‐based STDP

“…24 Complimentary to work on higher-precision efficient hardware implementation, as presented here, efforts on improving performance of low-precision networks have shown considerable progress recently. [25][26][27] Currently, these methods require off-chip processing during training and do not target online on-device learning in neuromorphic hardware.…”