F-PABEE: Flexible-Patience-Based Early Exiting For Single-Label and Multi-Label Text Classification Tasks

Gao, Xiaopeng; Zhu, Wei; Gao, Jian; Yin, Congrui

doi:10.1109/icassp49357.2023.10095864

Cited by 3 publications

(1 citation statement)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Full-model fine-tuning is one of the most widely used method for utilizing PTMs. However, fine-tuning (Devlin et al, 2019;Zhu et al, , 2021aZhu, 2021a;Gao et al, 2023;Zhang et al, 2023a) needs to tune all parameters of PTMs for each task, resulting in large GPU memory and storage costs, especially for supersized PTMs (Brown et al, 2020;Wang et al, 2021a). Parameter-efficient tuning (PETuning) is a new fine-tuning paradigm that can reduce the adaptation costs of PTMs by only tuning a very small number of internal or additional parameters (Ding et al, 2022;Zhang et al, 2023b;.…”

Section: Introductionmentioning

confidence: 99%

SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning

Zhu,

Tan

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Prompt tuning prepends a soft prompt to the input embeddings or hidden states and only optimizes the prompt to adapt pretrained models (PTMs) to downstream tasks. The previous work manually selects prompt layers which are far from optimal and failed to exploit the potential of prompt tuning. In this work, we propose a novel framework, Selective Prompt Tuning (SPT), that learns to select the proper prompt layers by inserting a prompt controlled by a learnable probabilistic gate at each intermediate layer. We further propose a novel bi-level optimization framework, SPT-DARTS, that can better optimize the learnable gates and improve the final prompt tuning performances of the learned prompt layer settings. We conduct extensive experiments with ten benchmark datasets under the full-data and few-shot scenarios. The results demonstrate that our SPT framework can perform better than the previous state-of-the-art PETuning baselines with comparable or fewer tunable parameters.

show abstract