How Well Do Sparse ImageNet Models Transfer?

Iofinova, Eugenia; Peşte, Alexandra; Kurtz, Mark; Alistarh, Dan

doi:10.1109/cvpr52688.2022.01195

Cited by 8 publications

(1 citation statement)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, one notable consideration is that our study focused on "leading accuracy" metrics, such as perplexity, which is essentially standard in the literature [5,34]. We believe a thorough study of the impact of compression upon secondary measures, and in particular transferrability [15] or bias effects [2] is warranted, and may be rendered easier through our work. At the same time, our work makes inference on extremely large language models more accessible, for better or for worse.…”

Section: Discussionmentioning

confidence: 99%

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Frantar¹,

Ashkboos²,

Hoefler³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generative Pre-trained Transformer (GPT) models set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs to execute, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highlyaccurate and highly-efficient. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU. We show experimentally that these improvements can be leveraged for endto-end inference speedups over FP16, of around 2x when using high-end GPUs (NVIDIA A100) and 4x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/gptq.

show abstract

Section: Discussionmentioning

confidence: 99%