Parameter-Efficient Transfer Learning with Diff Pruning

Abstract

The large size of pretrained networks makes them difficult to deploy for multiple tasks in storage-constrained settings. Diff pruning enables parameter-efficient transfer learning that scales well with new tasks. The approach learns a task-specific "diff" vector that extends the original pretrained parameters. This diff vector is adaptively pruned during training with a differentiable approximation to the L 0 -norm penalty to encourage sparsity. As the number of tasks increases, diff pruning remains parameter-efficient, as it requires storing only a small diff vector for each task. Since it does not require access to all tasks during training, it is attractive in on-device deployment settings where tasks arrive in stream or even from different providers. Diff pruning can match the performance of finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model's parameters per task and scales favorably in comparison to popular pruning approaches.

UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning

“…Alternatively, some methods fix the entire PLM and introduce a small number of new trainable parameters. Notable examples in this category include adapter-tuning (Houlsby et al, 2019) and its extensions (Pfeiffer et al, 2021;, prefix-tuning (Li and Liang, 2021) and its extensions (Lester et al, 2021), and additive methods (Zhang et al, 2020;Guo et al, 2021;Hu et al, 2021).…”

“…Additive PELT methods treat the model parameters after fine-tuning as an addition of the pre-trained parameters θ pre-trained and task-specific differences δ task , where θ pre-trained is fixed and a new (sub)set of model parameters are added on top (θ task = θ pre-trained + δ task ). There are various ways to parameterize the task-specific differences δ task , leading to different additive methods such as LoRA (Hu et al, 2021), diff pruning (Guo et al, 2021), and side-tuning (Zhang et al, 2020). We plan to incorporate additive methods into UNIPELT in the next version.…”

“…Adapter-tuning & extensions 0.047% ~3.6% Prefix-tuning & extensions 0.01% ~2% BitFit (Ben Zaken et al, 2021) 0.01% ~0.09% Diff pruning (Guo et al, 2021) 0.5% LoRA (Hu et al, 2021) 0.01% UNIPELT 0.17%+0.81%=0.98%…”

“…One line of work proposes to only tune a small subset of the parameters such as the top layers or the bias terms (Ben Zaken et al, 2021). Other studies take a step further by freezing the entire PLM and adding a small number of additional trainable parameters (Houlsby et al, 2019;Li and Liang, 2021;Lester et al, 2021;Guo et al, 2021;Hu et al, 2021).…”

“…Existing PELT research generally aims at achieving performance comparable to conventional finetuning with as few trainable parameters as possible, which has seen significant progress -the task-specific trainable parameters used in most recent approaches (Lester et al, 2021;Guo et al, 2021) are almost negligible compared to the total parameters of the PLM (<1%). A more challenging yet barely studied problem is whether one can achieve better performance than fine-tuning with fewer parameters.…”