Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey

Abstract

Vision Transformers (ViTs) have recently garnered considerable attention, emerging as a promising alternative to convolutional neural networks (CNNs) in several vision-related applications. However, their large model sizes and high computational and memory demands hinder deployment, especially on resource-constrained devices. This underscores the necessity of algorithm-hardware co-design specific to ViTs, aiming to optimize their performance by tailoring both the algorithmic structure and the underlying hardware accelerator to each other's strengths. Model quantization, by converting high-precision numbers to lower-precision, reduces the computational demands and memory needs of ViTs, allowing the creation of hardware specifically optimized for these quantized algorithms, boosting efficiency. This article provides a comprehensive survey of ViTs quantization and its hardware acceleration. We first delve into the unique architectural attributes of ViTs and their runtime characteristics. Subsequently, we examine the fundamental principles of model quantization, followed by a comparative analysis of the state-of-the-art quantization techniques for ViTs. Additionally, we explore the hardware acceleration of quantized ViTs, highlighting the importance of hardware-friendly algorithm design. In conclusion, this article will discuss ongoing challenges and future research paths. We consistently maintain the related opensource materials at https://github.com/DD-DuDa/awesome-vitquantization-acceleration.

MOSDA: On-Chip Memory Optimized Sparse Deep Neural Network Accelerator With Efficient Index Matching

“…As DNN networks become deeper and more complex, the required computing power and energy consumption are also increasing [7]- [9]. Since most endpoint devices are batterypowered, energy-efficient ASICs which are able to process DNN is highly required.…”

“…Index matching is to match the coordinate of the weight and the coordinate of the input feature map (ifmap) pixel for each operation to determine whether there is a meaningful operation that produces a non-zero product term. However, the irregular distribution of non-zero data leads to a large matching overhead in parallel processing since accelerators have to match multiple coordinates in parallel [9].…”

Recent Progress on Memristive Convolutional Neural Networks for Edge Intelligence

“…[ 106 ] The error from each layer might accumulate and cause accuracy loss or nonconvergence. [ 144 ] In addition, end‐to‐end network adaptation for array nonideal factors is an interesting line of thinking. Hybrid training takes more account of the energy and complexity of a mapping device.…”

Brain-Inspired Computing: Models and Architectures

“…There have also been significant strides in the development of hardware accelerators for SNNs [116], [117], [118], CNNs [119], [120], [121], GNNs [122], [123] and training accelerators [124], [125], [126]. A comprehensive survey of the topic can be found in [127], [121]. We also refer the reader to recent research on attention networks [128] used in image captioning applications, transformers [129] used in natural language processing and on neural architecture search [130] to design neural network configurations that reduce the complexity of the network.…”