Ask Me Anything: A simple strategy for prompting language models

Abstract

Large language models (LLMs) transfer well to new tasks out-of-the-box simply given a natural language prompt that demonstrates how to perform the task and no additional training. Prompting is a brittle process wherein small modifications to the prompt can cause large variations in the model predictions, and therefore significant effort is dedicated towards designing a painstakingly perfect prompt for a task. To mitigate the high degree of effort involved in prompting, we instead ask whether collecting multiple effective, yet imperfect, prompts and aggregating them can lead to a high quality prompting strategy. Our observations motivate our proposed prompting method, ASK ME ANYTHING PROMPTING (AMA). We first develop an understanding of the effective prompt formats, finding question-answering (QA) prompts, which encourage open-ended generation ("Who went to the park?") tend to outperform those that restrict the model outputs ("John went to the park. Output True or False"). Our approach recursively uses the LLM to transform task inputs to the effective QA format. We apply these prompts to collect several noisy votes for the input's true label. We find that these prompts can have very different accuracies and complex dependencies and thus propose to use weak supervision, a procedure for combining the noisy predictions, to produce the final predictions. We evaluate AMA across open-source model families (EleutherAI, BLOOM, OPT, and T0) and sizes (125M-175B parameters), demonstrating an average performance lift of 10.2% over the few-shot baseline. This simple strategy enables the open-source GPT-J-6B model to match and exceed the performance of few-shot GPT3-175B on 15 of 20 popular benchmarks. Averaged across these tasks, the GPT-J-6B model outperforms few-shot GPT3-175B. We release our code for reproducing the results here: https://github.com/HazyResearch/ama_prompting.Recent work has evaluated LLM prompting performance on a broad set of tasks and finds the process to be brittlesmall changes to the prompt result in large performance variations [Zhao et al., 2021, Holtzman et al., 2021. The performance further varies depending on the chosen LLM family [Ouyang et al., 2022 and model size [Wei et al., 2022a, Lampinen et al., 2022. To improve reliability, significant effort is dedicated towards designing a painstakingly perfect prompt. For instance, Mishra et al. [2021] and recommend that users manually explore large search-spaces of strategies to tune their prompts on a task-by-task basis.

AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning

“…Crowdworkers have different viewpoints [6,24,83] and sourcing perspectives from expert crowdworkers [27] or from workers with various skill levels [102] can extend this variety. For LLMs, leveraging their stochastic nature (e.g., with a higher temperature parameter setting) or using different models or prompt variants (including role-based prompts) can source a variety of responses [12,117,143]. In some cases one can automatically calculate the optimal mix of LLMs for a task [100].…”

“…Simplifying tasks can increase quality [28,83,120,142]. Another strategy is to adapt the subtask to fit the worker's capabilities [12,23,109,120]. For example, researchers found that crowdworkers were better at generating predictive features than at estimating if a feature is predictive; so they adapted the workflow accordingly [25].…”

“…ideas. Despite their widespread adoption, LLMs suffer from quality deficits like hallucinations [40,106,111], brittleness to prompt changes [12,45,55,114,152], and user interventions limited to prompt changes [116,143]. These deficits are exacerbated for complex tasks, which require multiple steps in which errors can arise and propagate [39].…”

Assessing GPT-4’s Role as a Co-Collaborator in Scientific Research: A Case Study Analyzing Einstein’s Special Theory of Relativity

“…into the importance of prompt engineering, the development of frameworks and strategies for structuring prompts, and the evaluation of LLM performance on complex tasks. 5,6,[24][25][26][27][28][29] These studies highlight the significance of prompt engineering when working with models like GPT-4 and provide relevant context for using LLMs in advanced research, such as the present study on SRT. This work differs from these studies in that it specifically evaluates GPT-4, which is posited to possess emergent, foundational AGI behaviors.…”

Hyena Hierarchy: Towards Larger Convolutional Language Models

“…SuperGLUE: We evaluate models on the SuperGLUE (Wang et al, 2019) with the parsing pipeline of (Arora et al, 2022). For all tasks except WIC, CB and BoolQ, we generate a response using greedy decoding, then check for the gold label.…”