Prompt-free and Efficient Few-shot Learning with Language Models

Abstract

Current methods for few-shot fine-tuning of pretrained masked language models (PLMs) require carefully engineered prompts and verbalizers for each new task to convert examples into a cloze-format that the PLM can score. In this work, we propose PERFECT, a simple and efficient method for few-shot fine-tuning of PLMs without relying on any such handcrafting, which is highly effective given as few as 32 data points. PERFECT makes two key design choices: First, we show that manually engineered task prompts can be replaced with task-specific adapters that enable sample-efficient fine-tuning and reduce memory and storage costs by roughly factors of 5 and 100, respectively. Second, instead of using handcrafted verbalizers, we learn new multi-token label embeddings during fine-tuning, which are not tied to the model vocabulary and which allow us to avoid complex auto-regressive decoding. These embeddings are not only learnable from limited data but also enable nearly 100x faster training and inference. Experiments on a wide range of few shot NLP tasks demonstrate that PERFECT, while being simple and efficient, also outperforms existing state-of-theart few-shot learning methods. Our code is publicly available at https://github.com/ facebookresearch/perfect.git.

Can Foundation Models Help Us Achieve Perfect Secrecy?

“…For the language tasks, we use PERFECT, a recent adaptation method from Mahabadi et al [48] 9 , which inserts an adapter layer after the feed-forward block of each transformer layer of a RoBERTa-Large model, an early FM consisting of 355M parameters. This results in 3.3M trainable parameters.…”

“…Foundation Models We use popular existing FMs and FM adaptation techniques in our evaluations. In particular, we use simple zero-shot [58], in-context learning [10] and lightweight tuning [29,48] methods (defined in Section 3). We are inspired by work on versatile FM systems with natural language interfaces [37,77], though instead of ML methods, our focus is the consequences of these FM capabilities on privacy.…”

“…Several recent works propose to fine-tune a small number of parameters (a subset of the existing FM or newly initialized) [29,30,38]. We specifically require lightweight tuning approaches that suffice with few-samples [48,43].…”

“…Our goal is to understand if FMs can be applied to achieve strong privacy guarantees for personal tasks. We study the behavior of these nascent models in the standard FL setting using popular off-the-shelf FMs [56,72] and methods for FM adaptation [10,29,48,58]. We propose Foundation model Controls for User Secrecy (FOCUS), a framework for privately serving personal tasks, that consists of a unidirectional data flow architecture.…”

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

“…Mahabadi et al [28] found that PEFT can outperform standard fine-tuning in the low-resource setting. In concurrent work, Mahabadi et al [76] compare PEFT to the use of discrete prompts (e.g. PET [70]) during few-shot fine-tuning and find that PEFT compares favorably.…”