Extreme Compression for Pre-trained Transformers Made Simple and Efficient

Abstract

Extreme compression, particularly ultra-low bit precision (binary/ternary) quantization, has been proposed to fit large NLP models on resource-constraint devices. However, to preserve the accuracy for such aggressive compression schemes, cutting-edge methods usually introduce complicated compression pipelines, e.g., multi-stage expensive knowledge distillation with extensive hyperparameter tuning. Also, they oftentimes focus less on smaller transformer models that have already been heavily compressed via knowledge distillation and lack a systematic study to show the effectiveness of their methods. In this paper, we perform a very comprehensive systematic study to measure the impact of many key hyperparameters and training strategies from previous works. As a result, we find out that previous baselines for ultra-low bit precision quantization are significantly under-trained. Based on our study, we propose a simple yet effective compression pipeline for extreme compression, named XTC. XTC demonstrates that (1) we can skip the pre-training knowledge distillation to obtain a 5-layer BERT while achieving better performance than previous state-of-the-art methods, e.g., the 6-layer TinyBERT; (2) extreme quantization plus layer reduction is able to reduce the model size by 50x, resulting in new state-of-the-art results on GLUE tasks. * Equal contribution. Code will be released soon as a part of https://github.com/microsoft/DeepSpeed

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

“…In addition, our study focuses on generative tasks, and does not consider activation quantization, nor speedups in batched execution. These are natural directions for future work, and we believe this can be achieved with carefully-designed GPU kernels and extensions of existing complementary techniques [34,33].…”

“…To our knowledge, we are the first to show that extremely accurate language models with hundreds of billions of parameters can be quantized to 2.5 − 4 bits per component on average: prior post-training methods only remain accurate at 8 bits [34,5], while prior training-based techniques have only tackled models that are smaller by one to two orders of magnitude [33]. This high degree of compression may appear unsurprising, as these networks are overparametrized; yet, as we discuss in our detailed analysis of results, compression induces non-trivial tradeoffs between the accuracy of the language modeling (perplexity), bit-width, and the size of the original model.…”

Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

“…However, most of those works need quantization-aware finetuning or even expensive quantizationaware knowledge distillation (Hinton, Vinyals, and Dean 2014). Due to the cost of training/finetuning LLMs (Polino, Pascanu, and Alistarh 2018;Jiao et al 2019;Tao et al 2022;Wu et al 2022Wu et al , 2023, it is a challenge for practitioners/researchers to do finetuning/distillation on those LLMs, particularly for models like GPT-3-175B (Brown et al 2020) and BLOOM-176B (Scao et al 2022).…”