SPIQ: Data-Free Per-Channel Static Input Quantization

Abstract

Computationally expensive neural networks are ubiquitous in computer vision and solutions for efficient inference have drawn a growing attention in the machine learning community. Examples of such solutions comprise quantization, i.e. converting the processing values (weights and inputs) from floating point into integers e.g. int8 or int4. Concurrently, the rise of privacy concerns motivated the study of less invasive acceleration methods, such as data-free quantization of pre-trained models weights and activations. Previous approaches either exploit statistical information to deduce scalar ranges and scaling factors for the activations in a static manner, or dynamically adapt this range on-the-fly for each input of each layers (also referred to as activations): the latter generally being more accurate at the expanse of significantly slower inference. In this work, we argue that static input quantization can reach the accuracy levels of dynamic methods by means of a per-channel input quantization scheme that allows one to more finely preserve cross-channel dynamics. We show through a thorough empirical evaluation on multiple computer vision problems (e.g. ImageNet classification, Pascal VOC object detection as well as CityScapes semantic segmentation) that the proposed method, dubbed SPIQ, achieves accuracies rivalling dynamic approaches with static-level inference speed, significantly outperforming state-of-the-art quantization methods on every benchmark.Preprint. Under review.

SPIQ: Data-Free Per-Channel Static Input Quantization

“…In Table 3, we report our extensive study of post-training W4/A4 quantization techniques on convolutional neural networks (ResNets, MobileNets and EfficientNets) as well as transformers from ViT b16 (86M parameters) to ViT h14 (600M parameters). In this extreme compression regime, we observe the limits of previous state-of-the art methods SQuant [8] and SPIQ [45]. This is not the case for Pow-erQuant which already achieves strong results on ResNets and transformers and, as such, offers a very strong baseline for the proposed NUPES method.…”

“…In Table 5, we report our results for several large language models on common sense reasoning tasks. We do not use group-wise quantization as it leads to incompatibility with activation quantization due to the constraint of dimensionality as explained in SPIQ [45]. In other words, while we can demonstrate that group-wise quantization can lead to higher compression rate for the weights, such methods are bound to never quantize the activations.…”

“…In its most practical setup from the user perspective, but most challenging in terms of compression, quantization only leverages the weight values of a pre-trained model. Data-free quantization techniques [8], [35], [45], [51] are motivated by the growing concerns regarding data privacy as well as their scalability with respect to the model size. On the other end of the spectrum, quantization aware training techniques [9], [10], [50] representations at the cost of a full retraining burdened by extra operations.…”

“…As opposed to most quantization methods [28], [35], [45], [50] which, for the sake of practicality, map floating point values to an evenly spread, discrete space, non-uniform quantization achieves a tighter fit to the target distribution by using a non-uniform spread quantized target space. However such methods [2], [33] require custom implementations that significantly shift away from uniform quantization.…”