Structured Pruning of Large Language Models

Abstract

Large language models have recently achieved state of the art performance across a wide variety of natural language tasks. Meanwhile, the size of these models and their latency have significantly increased, which makes their usage costly, and raises an interesting question: do language models need to be large? We study this question through the lens of model compression. We present a generic, structured pruning approach by parameterizing each weight matrix using its low-rank factorization, and adaptively removing rank-1 components during training. On language modeling tasks, our structured approach outperforms other unstructured and block-structured pruning baselines at various compression levels, while achieving significant speedups during both training and inference. We also demonstrate that our method can be applied to pruning adaptive word embeddings in large language models, and to pruning the BERT model on several downstream fine-tuning classification benchmarks. 1 * Denotes equal contribution.

Which *BERT? A Survey Organizing Contextualized Encoders

“…Another class of approaches carefully selects weights to reduce model size. Lan et al (2020) use low-rank factorization to reduce the size of the embedding matrices, while Wang et al (2019f) factorize other weight matrices. Additionally, parameters can be shared between layers (Dehghani et al, 2019;Lan et al, 2020) or between an encoder and decoder (Raffel et al, 2019).…”

Parameter-Efficient Transfer Learning with Diff Pruning

“…This objective is still complicated by the discrete nature of z τ 's, but the expectation provides some guidance for empirically effective relaxations. We follow prior work (Louizos et al, 2018;Wang et al, 2019b) and relax z τ into continuous space [0, 1] d with a stretched Hard-Concrete distribution (Jang et al, 2017;Maddison et al, 2017), which allows for the use of pathwise gradient estimators. Specifically, z τ is now defined to be a deterministic and (sub)differentiable function of a sample u from a uniform distribution,…”

“…For evaluation we use the GLUE benchmark (Wang et al, 2019b) as well as the SQuAD extractive question answering dataset (Rajpurkar et al, 2016 Devlin et al (2019) to compare against the adapter-based approach of Houlsby et al (2019). Our implementation is based on the Hugging Face Transformer library .…”

“…Structured diff pruning introduces an additional mask per group, which encourages pruning of entire groups. This is less restrictive than traditional group sparsity techniques that have been used with L 0 -norm relaxations, which force all parameters in a group to share the same mask (Louizos et al, 2018;Wang et al, 2019b). However we still expect entire groups to be pruned out more often, which might bias the learning process towards either eliminating completely or clustering together nonzero diffs.…”

The NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures

“…To overcome the above shortcomings, a novel structured pruning paradigm was introduced [78] with low-rank factorization which retained the dense matrix structure and 0 norm which relaxed constraints enforced via structured pruning. The weight matrices were factorized into a product of two smaller matrices with a diagonal mask that was pruned while training via 0 regularizer that controlled the end sparsity of the model.…”