OMNI: A Framework for Integrating Hardware and Software Optimizations for Sparse CNNs

Abstract

Convolution neural networks (CNNs) as one of today's main flavor of deep learning techniques dominate in various image recognition tasks. As the model size of modern CNNs continues to grow, neural network compression techniques have been proposed to prune the redundant neurons and synapses. However, prior techniques disconnect the software neural networks compression and hardware acceleration, which fail to balance multiple design parameters including sparsity, performance, hardware area cost, and efficiency. More concretely, prior unstructured pruning techniques achieve high sparsity at the expense of extra performance overhead, while prior structured pruning techniques relying on strict sparse patterns lead to low sparsity and extra hardware cost. In this paper, we propose OMNI, a framework for accelerating sparse CNNs on hardware accelerators. The innovation of OMNI stems from that it uses hardware amenable on-chip memory partition patterns to seamlessly engage the software CNN model compression and hardware CNN acceleration. To accelerate the compute-intensive convolution kernel, a promising hardware optimization approach is memory partition, which divides the original weight kernels into several groups so that the different hardware processing elements can simultaneously access the weight. We exploit the memory partition patterns including block, cyclic, or hybrid as a means of CNN compression patterns. Our software CNN model compression balances the sparsity across different groups and our hardware accelerator employs hardware parallelization coordinately with the sparse patterns, leading to a desirable compromise between sparsity and performance. We further develop performance models to help the designers to quickly identify the pattern factors subject to an area constraint. Last, we evaluate our design on Application Specific Integrated Circuit (ASIC) and Field Programmable Gate Array (FPGA) platform. Experiments demonstrate that OMNI achieves 3.4x-6.2x speedup for the modern CNNs, over a comparably ideal dense CNN accelerator. OMNI shows 114.7x energy efficiency improvement compared with GPU platform. OMNI is also evaluated on Xilinx ZC706 and ZCU102 FPGA platforms, achieving 41.5 GOP/s and 125.3 GOP/s, respectively.

Morphling: A Reconfigurable Architecture for Tensor Computation

“…Sparse tensor accelerators. [13,25,26,44,50,51,89,92] are sparse DNN accelerators. MAERI [41] uses tree-based interconnects for data distribution and reduction which is similar to our reconfigurable adder tree.…”

Spatial Data Dependence Graph Based Pre-RTL Simulator for Convolutional Neural Network Dataflows

“…For example, the latency of MRAM tends to be substantially larger than that of SRAM latency [58]. Moreover, the bandwidth of local memory also varies between memory blocks, depending on the number of banks allocated to those blocks (e.g., [19] and [59,60]). Thus, each PE tends to experience an order-of-magnitude difference in its latency and bandwidth, depending on which memory block the activations (or filters) are transferred from/to.…”

An Efficient Hardware Design for Accelerating Sparse CNNs With NAS-Based Models

“…Most of these works optimize their dataflows based on loop operations like loop interchange and loop unrolling [16][17][18][19][20]. The dense accelerator can result in high hardware inefficiency since most multiplication operations involve zero operands [5,6,16,[21][22][23][24][25]. Implementation of sparse DNNs has been studied in recent years on FPGAs [26].…”