Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

Abstract

When primed with only a handful of training samples, very large, pretrained language models such as GPT-3 have shown competitive results when compared to fully-supervised, finetuned, large, pretrained language models. We demonstrate that the order in which the samples are provided can make the difference between near state-of-the-art and random guess performance: essentially some permutations are "fantastic" and some not. We analyse this phenomenon in detail, establishing that: it is present across model sizes (even for the largest current models), it is not related to a specific subset of samples, and that a given good permutation for one model is not transferable to another. While one could use a development set to determine which permutations are performant, this would deviate from the true fewshot setting as it requires additional annotated data. Instead, we use the generative nature of language models to construct an artificial development set and based on entropy statistics of the candidate permutations on this set, we identify performant prompts. Our method yields a 13% relative improvement for GPTfamily models across eleven different established text classification tasks.

Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

“…Prompt construction requires a non-trivial combinatorial search over the prompt's wording, whether to include training examples, and how to convert LM probabilities to class predictions. As a consequence, prompts are either designed using human intuition that is hard to replicate and apply in a principled manner (Perez et al, 2021), or using automated methods (Shin et al, 2020;Gao et al, 2021;Lu et al, 2021). These methods search for elements such as: (1) the text of the pattern, (2) the tokens in the verbalizers, and (3) whether and how training examples are prepended before the test input.…”

“…Thus far, we have shown that prompt-based finetuning can simplify prompt engineering at the cost of memory inefficiency-a new set of parameters must be learned for each task. This is in contrast to in-context learning, which holds all model weights fixed but is heavily influenced by small prompt modifications (Zhao et al, 2021;Lu et al, 2021). In this section, we investigate how to achieve both memory efficiency and simple prompts.…”

“…In stateof-the-art NLP, few-shot learning is performed by reformulating tasks as natural language "prompts" and completing those prompts with pre-trained language models (Brown et al, 2020;Schick and Schütze, 2021a). Prompts that are well-designed can substantially improve accuracy (Zhao et al, 2021;Lu et al, 2021). However, finding these prompts is difficult: it requires a non-trivial combinatorial search over the prompt's wording (a.k.a.…”

Few-shot Mining of Naturally Occurring Inputs and Outputs

“…Our method provides a way to collect large amounts of training data using a small set of labeled seed examples, and allows for more direct control over what the model learns compared to relatively brittle prompts (Lu et al, 2021). Yet, unlike model Summary: The Scottish city of Edinburgh is looking to crack down on so-called "silent disco" walking tours as residents complain they make too much noise.…”

Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning

“…Notably, pre-trained language models (PLMs) have learned a substantial amount of in-depth knowledge from data, and have archived tremendous promise in few-shot/zero-shot learning ability with the natural language prompts [12,48,54]. However, Recent studies [35,37,56] observe that prompt learning with PLMs usually generalizes unstably in an extremely low-resource setting or emerging domains. One potential reason is that, it is non-trivial for parametric models to learn rare or hard patterns well with rote memorization, thus, resulting in inefficient generalizable performance.…”