On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

He, Ruidan; Liu, Linlin; Hai, Yang; Tan, Qingyu; Ding, Bosheng; Cheng, Liying; Low, Jia-Wei; Si, Luo

doi:10.48550/arxiv.2106.03164

Cited by 11 publications

(14 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are two series of related works: invasive methods and non-invasive methods. Invasive methods, which are built on a strong assumption that the inner structure (e.g., self-attention and feedforward layers) of the PLM can be modified, includes Prefix-Tuning (Li and Liang, 2021), Bitfit (Ben Zaken et al, 2021), Child-Tuning , P-Tuning v2 (Liu et al, 2021b), LoRA (Hu et al, 2021, UnifiedSKG (Xie et al, 2022) and Adapter-based models (Rebuffi et al, 2017;Houlsby et al, 2019;Lin et al, 2020;He et al, 2021;Pfeiffer et al, 2021). Non-invasive methods, which only modify input embeddings and regard the inner structure as a black box, mostly are prompting methods (including our Input-Tuning).…”

Section: Related Workmentioning

confidence: 99%

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

An¹,

Li²,

Lin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently the prompt-tuning paradigm has attracted significant attention. By only tuning continuous prompts with a frozen pretrained language model (PLM), prompt-tuning takes a step towards deploying a shared frozen PLM to serve numerous downstream tasks. Although prompt-tuning shows good performance on certain natural language understanding (NLU) tasks, its effectiveness on natural language generation (NLG) tasks is still underexplored. In this paper, we argue that one of the factors hindering the development of prompt-tuning on NLG tasks is the unfamiliar inputs (i.e., inputs are linguistically different from the pretraining corpus). For example, our preliminary exploration reveals a large performance gap between prompt-tuning and fine-tuning when unfamiliar inputs occur frequently in NLG tasks. This motivates us to propose input-tuning, which fine-tunes both the continuous prompts and the input representations, leading to a more effective way to adapt unfamiliar inputs to frozen PLMs. Our proposed input-tuning is conceptually simple and empirically powerful. Experimental results on seven NLG tasks demonstrate that input-tuning is significantly and consistently better than prompt-tuning. Furthermore, on three of these tasks, input-tuning can achieve a comparable or even better performance than fine-tuning.

show abstract

Section: Related Workmentioning

confidence: 99%

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

An¹,

Li²,

Lin³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Because models are more prone to overfitting in low-data regimes, we hypothesize that, with more parameters available to the model, full fine-tuning and prefix tuning with special token embeddings overfit the training data and produce worse results on test data. Furthermore, according to He et al [8], the lower layers of a PLM capture more generic features while upper layers capture more task-specific features. Therefore, as the lowest layer in a PLM, if the special token embeddings in the embedding layer come to overfit the training data, the model will not be able to generalize well to unseen data.…”

Section: Low Data Settingmentioning

confidence: 99%

Unfreeze with Care: Space-Efficient Fine-Tuning of Semantic Parsing Models

Sun,

Khan,

Mesnards

et al. 2022

Preprint

View full text Add to dashboard Cite

Semantic parsing is a key NLP task that maps natural language to structured meaning representations. As in many other NLP tasks, SOTA performance in semantic parsing is now attained by finetuning a large pretrained language model (PLM). While effective, this approach is inefficient in the presence of multiple downstream tasks, as a new set of values for all parameters of the PLM needs to be stored for each task separately. Recent work has explored methods for adapting PLMs to downstream tasks while keeping most (or all) of their parameters frozen. We examine two such promising techniques, prefix tuning and bias-term tuning, specifically on semantic parsing. We compare them against each other on two different semantic parsing datasets, and we also compare them against full and partial fine-tuning, both in few-shot and conventional data settings. While prefix tuning is shown to do poorly for semantic parsing tasks off the shelf, we modify it by adding special token embeddings, which results in very strong performance without compromising parameter savings. CCS CONCEPTS• Computing methodologies → Natural language processing; Neural networks.

show abstract

“…The adapter's effectiveness was demonstrated over 26 diverse text classification tasks, where near full fine-tuning performance was achieved with only 3.6% of additional parameters per task. Later, He et al [4] showed that the adapter FT can mitigate forgetting issues of full FT because of the smaller weight deviations. LoRA [6] is another example of adapter-based LFT where augmentation is implemented with a different design of small weight modules.…”

Section: 21mentioning

confidence: 99%

Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning

Jung,

Choi,

Rhee

2021

Preprint

View full text Add to dashboard Cite

A BERT-based Neural Ranking Model (NRM) can be either a crossencoder or a bi-encoder. Between the two, bi-encoder is highly efficient because all the documents can be pre-processed before the actual query time. Although query and document are independently encoded, the existing bi-encoder NRMs are Siamese models where a single language model is used for consistently encoding both of query and document. In this work, we show two approaches for improving the performance of BERT-based bi-encoders. The first approach is to replace the full fine-tuning step with a lightweight fine-tuning. We examine lightweight fine-tuning methods that are adapter-based, prompt-based, and hybrid of the two. The second approach is to develop semi-Siamese models where queries and documents are handled with a limited amount of difference. The limited difference is realized by learning two lightweight fine-tuning modules, where the main language model of BERT is kept common for both query and document. We provide extensive experiment results for monoBERT, TwinBERT, and ColBERT where three performance metrics are evaluated over Robust04, ClueWeb09b, and MS-MARCO datasets. The results confirm that both lightweight fine-tuning and semi-Siamese are considerably helpful for improving BERT-based bi-encoders. In fact, lightweight fine-tuning is helpful for crossencoder, too. 1

show abstract

On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

Cited by 11 publications

References 15 publications

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

Unfreeze with Care: Space-Efficient Fine-Tuning of Semantic Parsing Models

Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning

Contact Info

Product

Resources

About