Diversifying Sample Generation for Accurate Data-Free Quantization

Zhang, Xiangguo; Qin, Haotong; Ding, Yifu; Gong, Ruihao; Yan, Qinghua; Tao, Renshuai; Li, Yuhang; Yu, Fei; Liu, Xianglong

doi:10.1109/cvpr46437.2021.01540

Cited by 59 publications

(35 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In cases such as binary or ternary quantization, proposed method would greatly benefit from fine-tuning. Generated data, obtained with similar methods as [41,40], may provide better insight on inputs and activation distributions, which could in theory improve the scales estimation for input quantization.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

SPIQ: Data-Free Per-Channel Static Input Quantization

Yvinec¹,

Dapogny²,

Cord³

et al. 2022

Preprint

View full text Add to dashboard Cite

Computationally expensive neural networks are ubiquitous in computer vision and solutions for efficient inference have drawn a growing attention in the machine learning community. Examples of such solutions comprise quantization, i.e. converting the processing values (weights and inputs) from floating point into integers e.g. int8 or int4. Concurrently, the rise of privacy concerns motivated the study of less invasive acceleration methods, such as data-free quantization of pre-trained models weights and activations. Previous approaches either exploit statistical information to deduce scalar ranges and scaling factors for the activations in a static manner, or dynamically adapt this range on-the-fly for each input of each layers (also referred to as activations): the latter generally being more accurate at the expanse of significantly slower inference. In this work, we argue that static input quantization can reach the accuracy levels of dynamic methods by means of a per-channel input quantization scheme that allows one to more finely preserve cross-channel dynamics. We show through a thorough empirical evaluation on multiple computer vision problems (e.g. ImageNet classification, Pascal VOC object detection as well as CityScapes semantic segmentation) that the proposed method, dubbed SPIQ, achieves accuracies rivalling dynamic approaches with static-level inference speed, significantly outperforming state-of-the-art quantization methods on every benchmark.Preprint. Under review.

show abstract

Section: Discussionmentioning

confidence: 99%

“…The standard method for input quantization, introduced in [30] and latter used in [8,41], is static as its parameters are computed once and for all during quantization then fixed during inference. This provides the best inference speed but for at the cost of a lower accuracy due to coarse modelization of the input ranges.…”

Section: Input Quantizationmentioning

confidence: 99%

SPIQ: Data-Free Per-Channel Static Input Quantization

Yvinec¹,

Dapogny²,

Cord³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To mitigate the accuracy degradation, almost all quantization methods require access to the original dataset for re-training/fine-tuning the model parameters [7,11,16,29,37]. Unfortunately, in scenarios involving sensitive data (e.g., medical and bio-metric data), these methods are no longer applicable due to the unavailability of the original dataset [34,38]. Therefore, data-free quantization is regarded as a potential and practice scheme and thus has recently been widely investigated [3,39].…”

Section: Introductionmentioning

confidence: 99%

“…The key issue is how to generate effective and meaningful samples to ensure the calibration accuracy. A notable line of research proposes batch normalization (BN) regularization [3,38], which states that the statistics (i.e., the mean and standard deviation) encoded in the BN layers can represent the distribution of original training data. These methods, however, are only applicable to convolutional neural networks (CNNs) and not to vision transformers, because the latter employs layer normalization (LN), which does not store any previous information like BN.…”

Section: Introductionmentioning

confidence: 99%

Patch Similarity Aware Data-Free Quantization for Vision Transformers

Li¹,

Ma²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Vision transformers have recently gained great success on various computer vision tasks; nevertheless, their high model complexity makes it challenging to deploy on resource-constrained devices. Quantization is an effective approach to reduce model complexity, and data-free quantization, which can address data privacy and security concerns during model deployment, has received widespread interest. Unfortunately, all existing methods, such as BN regularization, were designed for convolutional neural networks and cannot be applied to vision transformers with significantly different model architectures. In this paper, we propose PSAQ-ViT, a Patch Similarity Aware data-free Quantization framework for Vision Transformers, to enable the generation of "realistic" samples based on the vision transformer's unique properties for calibrating the quantization parameters. Specifically, we analyze the selfattention module's properties and reveal a general difference (patch similarity) in its processing of Gaussian noise and real images. The above insights guide us to design a relative value metric to optimize the Gaussian noise to approximate the real images, which are then utilized to calibrate the quantization parameters. Extensive experiments and ablation studies are conducted on various benchmarks to validate the effectiveness of PSAQ-ViT, which can even outperform the real-data-driven methods.

show abstract

“…Traditional standard CNN-based models contain millions of parameters [1,2,3,4,5], causing problems in many practical works, e.g., developing real-time interactive vision applications for portable devices, transferring knowledge from existing models to novel categories without re-training, etc. On the contrary, learning novel concepts from few examples is an interesting and challenging task with many practical advantages.…”

Section: Introductionmentioning

confidence: 99%

Multi-Pretext Attention Network for Few-shot Learning with Self-supervision

Tao

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Few-shot learning is an interesting and challenging study, which enables machines to learn from few samples like humans. Existing studies rarely exploit auxiliary information from large amount of unlabeled data. Self-supervised learning is emerged as an efficient method to utilize unlabeled data. Existing self-supervised learning methods always rely on the combination of geometric transformations for the single sample by augmentation, while seriously neglect the endogenous correlation information among different samples that is the same important for the task. In this work, we propose a Graph-driven Clustering (GC), a novel augmentationfree method for self-supervised learning, which does not rely on any auxiliary sample and utilizes the endogenous correlation information among input samples. Besides, we propose Multi-pretext Attention Network (MAN), which exploits a specific attention mechanism to combine the traditional augmentation-relied methods and our GC, adaptively learning their optimized weights to improve the performance and enabling the feature extractor to obtain more universal representations. We evaluate our MAN extensively on miniImageNet and tieredImageNet datasets and the results demonstrate that the proposed method outperforms the stateof-the-art (SOTA) relevant methods. 1

show abstract

Diversifying Sample Generation for Accurate Data-Free Quantization

Cited by 59 publications

References 27 publications

SPIQ: Data-Free Per-Channel Static Input Quantization

SPIQ: Data-Free Per-Channel Static Input Quantization

Patch Similarity Aware Data-Free Quantization for Vision Transformers

Multi-Pretext Attention Network for Few-shot Learning with Self-supervision

Contact Info

Product

Resources

About