Instruction Induction: From Few Examples to Natural Language Task Descriptions

Abstract

Large language models are able to perform a task by conditioning on a few inputoutput demonstrations -a paradigm known as in-context learning. We show that language models can explicitly infer an underlying task from a few demonstrations by prompting them to generate a natural language instruction that fits the examples. To explore this ability, we introduce the instruction induction challenge, compile a dataset consisting of 24 tasks, and define a novel evaluation metric based on executing the generated instruction. We discover that, to a large extent, the ability to generate instructions does indeed emerge when using a model that is both large enough and aligned to follow instructions; InstructGPT achieves 65.7% of human performance in our execution-based metric, while the original GPT-3 model reaches only 9.8% of human performance. This surprising result suggests that instruction induction might be a viable learning paradigm in and of itself, where instead of fitting a set of latent continuous parameters to the data, one searches for the best description in the natural language hypothesis space. 1 We examine instruction induction on 24 tasks, ranging from morphosyntactic tasks (e.g., pluralization) to style transfer (e.g., formality) and sentiment analysis. As a basic evaluation protocol, we collect human annotations and use them as gold-standard references; the generated instructions are then compared to these references using BERTScore [Zhang et al., 2020]. Moreover, we suggest a novel evaluation metric for instruction induction: execution accuracy. The execution accuracy of a generated instruction is measured by testing whether LMs can correctly perform the task in a zero-shot manner by using the generated instruction alone, without any demonstrations.1 Our code and data are publicly available at https://github.com/orhonovich/instruction-induction Preprint. Under review.

WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences

“…for guiding the LLM to generate a qualified answer. Recent work [9] suggests that we can take advantage of the LLM itself to design instructions for in-context learning instead of human handcrafting. We use several high-quality examples to induce a few possible instructions (Cf.…”

“…However, these instructions are too verbose, and in the presence of examples, the model's performance is not significantly impacted by the instructions. Therefore, we adopt a more natural approach to generate instructions [9] to produce a natural instruction that is interpretable by the model.…”

Task-Level Thinking Steps Help Large Language Models for Challenging Classification Task

“…In the era of LLMs, searching directly in the natural language hypothesis is more popular. Honovich et al (2022) found LLMs can generate the task instruction with several demonstrations. automatically generates many instructions for the given demonstrations and selects the one with the maximum score function.…”

MaNtLE: Model-agnostic Natural Language Explainer

“…Instruction Generation by Large Language Models. Some recent works (Honovich et al, 2022; datasets. While our training procedure learns to generate explanations for datasets akin to these prior works, our primary objective is to explain classifiers to understand their classification rationale rather than datasets.…”