Efficient Design Space Exploration for Sparse Mixed Precision Neural Architectures

Abstract

Pruning and Quantization are two effective Deep Neural Network (DNN) compression methods for efficient inference on various hardware platforms. Pruning refers to removing unimportant weights or nodes, whereas Quantization converts the floating-point parameters to low-bit fixed integer representation. The pruned and low precision models result in smaller and faster inference models on hardware platforms with almost the same accuracy as the unoptimized network. Tensor Cores in Nvidia Ampere 100 (A100) GPU supports (1) 2:4 fine-grained sparse pruning where 2 out of every 4 elements are pruned, and (2) traditional dense multiplication to achieve a good accuracy and performance trade-off. The A100 Tensor Core also takes advantage of 1-bit, 4-bit, and 8-bit multiplication to speed up the inference of a model. Hence, finding the right matrix type (dense or 2:4 sparse) along with the precision for each layer becomes a combinatorial problem. Neural Architecture Search (NAS) can alleviate such problems by automating the architecture design process instead of a brute-force search. In this paper, we propose (i) Mixed Sparse and Precision Search (MSPS), a NAS framework to search for efficient sparse and mixed-precision quantized model within the predefined search space and fixed backbone neural network (Eg. ResNet50), and (ii) Architecture, Sparse and Precision Search (ASPS) to jointly search for kernel size and number of filters, and sparse-precision combination of each layer. We illustrate the effectiveness of our methods targeting A100 Tensor Core on Nvidia GPUs by searching efficient sparse-mixed precision networks on ResNet50 and achieving better accuracy-latency trade-off models compared to the manually designed Uniform Sparse Int8 networks. CCS CONCEPTS• Computing methodologies → Neural networks.

Differentiable Neural Architecture, Mixed Precision and Accelerator Co-Search

“…The search space pruning can also be extended to other benchmark models. Our previous work [28] has shown a similar trend where different precisions could exhibit similar latency. 2) Skip Connection: We did not consider the skip connection in our Architecture and Mixed Precision search space.…”

“…The automatically searched mixed quantized networks offer better latency accuracy tradeoff than Uniform Quantization on CIFAR [26] and ImageNet [27] datasets. The search method is partly taken from our previous work [28], which we applied to a different hardware platform (Nvidia A100 GPU) and Neural Network (ResNet50). However, we significantly contributed to developing a new search space pruning and weight/activation sharing method in this paper.…”

Hardware-aware design, search, and optimization of deep neural networks

“…We develop Mixed Sparse and Precision Search (MSPS) technique[Chitty-Venkata et al (2022a)] to search for efficient weight matrix (dense or sparse) and precision combination for every layer on a fixed pretrained model (Section 6.4). The automatically generated MSPS networks outperform Uniform 2:4 Sparse Int8 and 4 configured networks in terms of accuracy and latency on CIFAR[Krizhevsky et al (2009)] and ImageNet[Deng et al (2009)] datasets.3.…”

“…The automatically generated MSPS networks outperform Uniform 2:4 Sparse Int8 and 4 configured networks in terms of accuracy and latency on CIFAR[Krizhevsky et al (2009)] and ImageNet[Deng et al (2009)] datasets.3. We extend MSPS and develop a technique to search for Neural Architecture, Sparsity pattern, and Precision (ASPS)[Chitty-Venkata et al (2022a)] to jointly optimize the macro-architecture (kernel size, number of filters) and sparse-precision combination of each layer (Section 6.5). The resulting ASPS outperforms both the baseline Uniform Sparse Int8…”