Structured Pruning Learns Compact and Accurate Models

Abstract

The growing size of neural language models has led to increased attention in model compression.The two predominant approaches are pruning, which gradually removes weights from a pre-trained model, and distillation, which trains a smaller compact model to match a larger one. Pruning methods can significantly reduce the model size but hardly achieve large speedups as distillation. However, distillation methods require large amounts of unlabeled data and are expensive to train. In this work, we propose a task-specific structured pruning method CoFi 1 (Coarse-and Fine-grained Pruning), which delivers highly parallelizable subnetworks and matches the distillation methods in both accuracy and latency, without resorting to any unlabeled data. Our key insight is to jointly prune coarse-grained (e.g., layers) and fine-grained (e.g., heads and hidden units) modules, which controls the pruning decision of each parameter with masks of different granularity. We also devise a layerwise distillation strategy to transfer knowledge from unpruned to pruned models during optimization. Our experiments on GLUE and SQuAD datasets show that CoFi yields models with over 10× speedups with a small accuracy drop, showing its effectiveness and efficiency compared to previous pruning and distillation approaches. 2

AutoDisc: Automatic Distillation Schedule for Large Language Model Compression

“…For the sake of a corner case that all structures in a module are pruned, we skip the module by feeding the input as the output. While we can alternate to an quite recent pruning method [22] exploiting both coarse-grained and fine-grained strategies for state-of-the-art performance, we argue that our framework is agnostic to pruning methods and keep the pruning method simple.…”

“…A number of approaches have been proposed to identify a good structure at a scale, including dynamic search [12], layer dropping [21] and pruning [11]. In this work, we adopt pruning to assign structures A k to the candidates due to its known advantages in knowledge distillation [22]. Concretely, following previous work [11], the pruning starts with the least important parameters/features based on their importance scores, which are approximated by masking the parameterized structures.…”

“…The GLUE originally consists of two sequence classification tasks, SST-2 [26] and CoLA [27], with seven sequence-pair classification tasks, i.e., MRPC [28], STS-B [29], QQP, MNLI [30], QNLI [31], RTE [32] and WNLI [33]. We exclude WNLI and CoLA due to the evaluation inconsistency (in other words, compressed LMs get dramatically worse results while original LMs get much better ones as found out in [22]) and use the other seven tasks for evaluation. Following the work in BERT [1], we report F1 on MRPC and QQP, Spearman Correlation scores (Sp Corr) on STS-B, and Accuracy (Acc) on other tasks.…”

“…Model Pruning Inspired by the idea that not all parameters contribute equally to the overall performance of a model, model pruning [43] is widely adopted to waive the parameters with little impact. Model pruning spans from unstructured pruning [19,[44][45][46] to structured pruning [11,12,18,22,47]. Unstructured pruning prunes parameters at neuron level referring to parameter magnitude [43,44] or learning dynamics [45], while structured pruning [11,22] prunes parameters at module level relying on parameter sensitivity to performance.…”

“…Model pruning spans from unstructured pruning [19,[44][45][46] to structured pruning [11,12,18,22,47]. Unstructured pruning prunes parameters at neuron level referring to parameter magnitude [43,44] or learning dynamics [45], while structured pruning [11,22] prunes parameters at module level relying on parameter sensitivity to performance. Although unstructured pruning enjoys a finer-grained pruning, it can only fit specialized devices.…”