Ps and Qs: Quantization-Aware Pruning for Efficient Low Latency Neural Network Inference

Abstract

Efficient machine learning implementations optimized for inference in hardware have wide-ranging benefits, depending on the application, from lower inference latency to higher data throughput and reduced energy consumption. Two popular techniques for reducing computation in neural networks are pruning, removing insignificant synapses, and quantization, reducing the precision of the calculations. In this work, we explore the interplay between pruning and quantization during the training of neural networks for ultra low latency applications targeting high energy physics use cases. Techniques developed for this study have potential applications across many other domains. We study various configurations of pruning during quantization-aware training, which we term quantization-aware pruning, and the effect of techniques like regularization, batch normalization, and different pruning schemes on performance, computational complexity, and information content metrics. We find that quantization-aware pruning yields more computationally efficient models than either pruning or quantization alone for our task. Further, quantization-aware pruning typically performs similar to or better in terms of computational efficiency compared to other neural architecture search techniques like Bayesian optimization. Surprisingly, while networks with different training configurations can have similar performance for the benchmark application, the information content in the network can vary significantly, affecting its generalizability.

Experimental Implementation of a Neural Network Optical Channel Equalizer in Restricted Hardware Using Pruning and Quantization

“…On the other hand, math-intensive tensor operations executed on INT8 types can see up to a 16× speed-up compared to the same operations in FP32. Moreover, memory-limited operations could see up to a 4× speed-up compared to the FP32 version [22][23][24]41 . Therefore, in addition to pruning, we will reduce the precision of the weights and activations to further decrease the computational complexity of the equalizer, employing the technique known as integer quantization 41 .…”

“…The quantization process can occur after the training or during it. The first case is known as post-training quantization (PTQ) and the second one is the quantization aware training [22][23][24] . In PTQ, a trained model has its weight and activations quantified.…”

“…Traditionally, the network complexity arithmetic has been measured using the number of MAC operations. However, in terms of the DSP processing, the number BoPs is a more appropriate metric to describe the computational complexity of the model 22,45 , as different data types for the network weights and activations are used. Thus, in this work, we use BoPs to quantify the NN equalizer complexity.…”

“…Thus, the use of NN-based methods, while being, undoubtedly, promising and attractive, faces a major challenge in optical channel equalization, where the computational complexity emerges as an important limiting real-time deployment factor 10,12,20,21 . We note here that it is, of course, well known that some NN architectures can be simplified without significantly affecting their performance, thanks, e.g., to strategies such as pruning and quantization 19,20,[22][23][24][25] . However, their application in the experimental environment of the resource-restricted hardware has not been yet fully studied in the context of coherent optical channel equalization.…”

Charged Particle Tracking via Edge-Classifying Interaction Networks

“…4. Quantization-aware training [55,[59][60][61][62][63][64][65][66][67][68] using QKeras [56,69] or BreviTas [50,70], parameter pruning [71][72][73][74][75][76], and general hardware-algorithm codesign can significantly reduce the necessary FPGA resources by reducing the required bit precision and removing irrelevant operations. 5.…”