Two sparsities are better than one: unlocking the performance benefits of sparse–sparse networks

Abstract

In principle, sparse neural networks should be significantly more efficient than traditional dense networks. Neurons in the brain exhibit two types of sparsity; they are sparsely interconnected and sparsely active. These two types of sparsity, called weight sparsity and activation sparsity, when combined, offer the potential to reduce the computational cost of neural networks by two orders of magnitude. Despite this potential, today’s neural networks deliver only modest performance benefits using just weight sparsity, because traditional computing hardware cannot efficiently process sparse networks. In this article we introduce Complementary Sparsity, a novel technique that significantly improves the performance of dual sparse networks on existing hardware. We demonstrate that we can achieve high performance running weight-sparse networks, and we can multiply those speedups by incorporating activation sparsity. Using Complementary Sparsity, we show up to 100X improvement in throughput and energy efficiency performing inference on FPGAs. We analyze scalability and resource tradeoffs for a variety of kernels typical of commercial convolutional networks such as ResNet-50 and MobileNetV2. Our results with Complementary Sparsity suggest that weight plus activation sparsity can be a potent combination for efficiently scaling future AI models.

Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training

“…By contrast, using DN on RNNs is beneficial because the fully-connected RNNs are weight-memory bounded, and the energy saving brought by temporal sparsity is much larger for RNNs. [7] and [10] exploit both DN activation sparsity and weight sparsity to achieve impressive inference performance on hardware. Another method that can create sparsity in neural networks is conditional computation, or skipping operations.…”

EGRU: Event-based GRU for activity-sparse inference and learning

“…Activity sparsity in RNNs has been proposed previously in various forms [29,47,48], but only focusing on achieving it during inference. Conditional computation is a form of activity sparsity used in [17] to scale to 1 trillion parameters.…”

Editorial: Focus issue on machine learning for neuromorphic engineering

“…Further, Moraitis et al (2022) introduce an unsupervised local training algorithm based on a combination of Hebbian plasticity with a soft winner take all mechanism. On the hardware side, two studies introduce new methods to exploit sparsity on current hardware (graphics processing unit (GPU) and field programmable gate array (FPGA)) to improve inference efficiency through Complementary Sparsity (Turner et al 2022) and Procedural connectivity (Hunter et al 2022). Finally, on the application side, DeWolf et al (2023) introduce a welcome closed-loop benchmark control task based on a robotic arm simulated in the popular Mojoco platform to showcase the inherent power efficiency and low latency of event-based computation.…”

“…Although the sparse activity and connectivity of SNNs should result in a proportional reduction in computing requirements, the irregular patterns of neuron interconnectivity and activity reduce the expected gains on current hardware. Hunter et al (2022) address this problem by structuring the sparsity to match the requirements of the target hardware for implementing sparse activation-sparse connectivity networks on FPGA. This restructuring is achieved by overlaying multiple sparse matrices to form a single dense structure if no two sparse matrices contain non-zero elements at the same location.…”