Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization

Abstract

The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of "lottery tickets", and training a certain collection of them (i.e., a subnetwork) can match the performance of the full model. In this paper, we study such a collection of tickets, which is referred to as "winning tickets", in extremely over-parametrized models, e.g., pre-trained language models. We observe that at certain compression ratios, the generalization performance of the winning tickets can not only match but also exceed that of the full model. In particular, we observe a phase transition phenomenon: As the compression ratio increases, generalization performance of the winning tickets first improves then deteriorates after a certain threshold. We refer to the tickets on the threshold as "super tickets". We further show that the phase transition is task and model dependent -as the model size becomes larger and the training data set becomes smaller, the transition becomes more pronounced. Our experiments on the GLUE benchmark show that the super tickets improve single task fine-tuning by 0.9 points on BERT-base and 1.0 points on BERT-large, in terms of task-average score. We also demonstrate that adaptively sharing the super tickets across tasks benefits multi-task learning 1 .

Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency

“…Michel et al (2019) propose a simple gradient-based importance score to prune attention heads. Prasanna et al (2020); Liang et al (2021) extend to prune other components like the feedforward network of the Transformer (Vaswani et al, 2017). Wang et al (2020c) decompose the pre-trained model weights and apply L 0 regularization (Louizos et al, 2018) to regulate the ranks of decomposed weights.…”

“…Large-scale pre-trained monolingual language models like BERT (Devlin et al, 2019) and RoBERTa (Liu et al, 2019) have shown promising results in various NLP tasks while suffering from their large model size and high latency. Structured pruning has proven to be an effective approach to compressing and accelerating these large monolingual language models (Michel et al, 2019;Wang et al, 2020c;Prasanna et al, 2020;Liang et al, 2021), making them practical for real-world applications.…”

“…Algorithms There exists a broad spectrum of pruning algorithms (Hoefler et al, 2021), and it is impossible to test all of them considering the cost of pre-training. We focus on two pruning algorithms that have been studied the most in monolingual pretrained models: the regularization-based pruning (Louizos et al, 2018;Wang et al, 2020c) (and our improved version) and the gradient-based pruning (Michel et al, 2019;Prasanna et al, 2020;Liang et al, 2021) (See Section 4). We experimentally find that the simplest gradient-based pruning is more effective for XLM-R (See Section 5.2).…”

“…In this section, we briefly review the structure of XLM-R , a Transformer encoder (Vaswani et al, 2017) pre-trained by masked language modeling task (Devlin et al, 2019). We also revisit how conventional structured pruning algorithms are applied to Transformers by introducing additional gating variables and setting appropriate values to them (See Figure 1 and also Prasanna et al (2020); Liang et al (2021)). The XLM-R model consists of N layers.…”

Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training

“…You et al (2020) draw early-bird tickets (prune the original network) at an early stage of training, and only train the subnetwork from then on. Some recent works extend the LTH from random initialization to pre-trained initialization (Prasanna et al, 2020;Liang et al, 2021;Chen et al, 2021b). Particularly, find that WTs, i.e., subnetworks of the pre-trained BERT, derived from the pre-training task of MLM using IMP are universally transferable to the downstream tasks.…”