4-bit Conformer with Native Quantization Aware Training for Speech Recognition

Abstract

End-to-end automatic speech recognition (ASR) models have seen revolutionary quality gains with the recent development of large-scale universal speech models (USM). However, deploying these massive USMs is extremely expensive due to the enormous memory usage and computational cost. Therefore, model compression is an important research topic to fit USM-based ASR under budget in real-world scenarios. In this study, we propose a USM fine-tuning approach for ASR, with a low-bit quantization and N :M structured sparsity aware paradigm on the model weights, reducing the model complexity from parameter precision and matrix topology perspectives. We conducted extensive experiments with a 2-billion parameter USM on a largescale voice search dataset to evaluate our proposed method. A series of ablation studies validate the effectiveness of up to int4 quantization and 2:4 sparsity. However, a single compression technique fails to recover the performance well under extreme setups including int2 quantization and 1:4 sparsity. By contrast, our proposed method can compress the model to have 9.4% of the size, at the cost of only 7.3% relative word error rate (WER) regressions. We also provided in-depth analyses on the results and discussions on the limitations and potential solutions, which would be valuable for future studies.

Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models

“…Model compression has commonly been achieved through a number of methods such as sparsity pruning [6,10,11], low-bit quantization [12,13,14], knowledge distillation [15,16], and lowrank matrix factorization [17,18]. These techniques can typically be applied regardless of the model architecture which allows them to be generalized to different tasks.…”

“…However, without structured sparsity [19], the resulting model requires irregular memory access and without hardware support, memory usage and computation become inefficient. Quantization is typically applied to reduce model weights from 32-bit floating point values down to 8-bit integer values, and is also applied to lower quantization levels (i.e., 1-bit, 2-bit, or 4-bit [5,14]) and even mixed-precision quantization [20]. However, computations on low-bit quantization level models are not available on typical real-world hardware.…”