A Survey on In-context Learning

Qingxiu, Dong,; Dai, Damai; Zheng, Chuansheng; Wu, Zhiyong; Chang, Baobao; Sun, Xingming; Xu, Jun; Li, Lei; Sui, Zhifang

doi:10.48550/arxiv.2301.00234

Cited by 33 publications

(41 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LLMs like GPT-3 [1], OPT [44], and PaLM [5] demonstrate emergent abilities as model and corpus sizes increase [37]. These abilities are learned from demonstrations containing a few examples in the context, which is known as in-context learning [8]. To enable reasoning in LLMs, [38] propose Chain-of-Thought (CoT) prompting, which adds multiple reasoning steps to the input question.…”

Section: Related Workmentioning

confidence: 99%

“…As LLMs continue to grow in model parameters and training corpus size, they are revealing emergent abilities that allow them to learn to reason from just a few demonstration examples within a given context [37]. This paradigm of learning is referred to as in-context learning (ICL) [8]. Recently, approaches [42,13] Figure 1: Two approaches for solving the DIE task: (a) previous pre-trained document understanding models [14,41] have been proposed to explore how to use LLMs to solve vision-language (VL) tasks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction

He¹,

Wang²,

Hu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Large language models (LLMs), such as GPT-3 and ChatGPT, have demonstrated remarkable results in various natural language processing (NLP) tasks with incontext learning, which involves inference based on a few demonstration examples. Despite their successes in NLP tasks, no investigation has been conducted to assess the ability of LLMs to perform document information extraction (DIE) using in-context learning. Applying LLMs to DIE poses two challenges: the modality and task gap. To this end, we propose a simple but effective in-context learning framework called ICL-D3IE, which enables LLMs to perform DIE with different types of demonstration examples. Specifically, we extract the most difficult and distinct segments from hard training documents as hard demonstrations for benefiting all test instances. We design demonstrations describing relationships that enable LLMs to understand positional relationships. We introduce formatting demonstrations for easy answer extraction. Additionally, the framework improves diverse demonstrations by updating them iteratively. Our experiments on three widely used benchmark datasets demonstrate that the ICL-D3IE framework enables GPT-3/ChatGPT to achieve superior performance when compared to previous pretrained methods fine-tuned with full training in both the in-distribution (ID) setting and in the out-of-distribution (OOD) setting.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction

He¹,

Wang²,

Hu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Moreover, these models do not only exhibit remarkable performance on tasks they were trained for but also quickly adapt to other novel and complex tasks. This is made possible through a mechanism known as in-context learning, which allows these models to learn from a limited number of input and label pairs, commonly referred to as few-shot prompts [4], provided during test time. Prior research has also demonstrated that the performance of these models on sophisticated reasoning tasks can be significantly improved by presenting them with human-annotated rationales alongside input/label pairs during test time [5,6].…”

Section: Introductionmentioning

confidence: 99%

“…(3) We then use explainability techniques to compute explanations for the selected samples with respect to their ground truth labels. (4) We construct the few-shot prompt for LLM using the samples selected and their corresponding explanations to feed as input to LLM for prediction.…”

Section: Introductionmentioning

confidence: 99%

Post Hoc Explanations of Language Models Can Improve Language Models

Krishna¹,

Ma²,

Slack³

et al. 2023

Preprint

View full text Add to dashboard Cite

Large Language Models (LLMs) have demonstrated remarkable capabilities in performing complex tasks. Moreover, recent research has shown that incorporating human-annotated rationales (e.g., Chain-of-Thought prompting) during in-context learning can significantly enhance the performance of these models, particularly on tasks that require reasoning capabilities. However, incorporating such rationales poses challenges in terms of scalability as this requires a high degree of human involvement. In this work, we present a novel framework, Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY), which addresses the aforementioned challenges by automating the process of rationale generation. To this end, we leverage post hoc explanation methods which output attribution scores (explanations) capturing the influence of each of the input features on model predictions. More specifically, we construct automated natural language rationales that embed insights from post hoc explanations to provide corrective signals to LLMs. Extensive experimentation with real-world datasets demonstrates that our framework, AMPLIFY, leads to prediction accuracy improvements of about 10-25% over a wide range of tasks, including those where prior approaches which rely on human-annotated rationales such as Chain-of-Thought prompting fall short. Our work makes one of the first attempts at highlighting the potential of post hoc explanations as valuable tools for enhancing the effectiveness of LLMs. Furthermore, we conduct additional empirical analyses and ablation studies to demonstrate the impact of each of the components of AMPLIFY, which, in turn, lead to critical insights for refining in-context learning.

show abstract

“…The key idea of prompt engineering is to provide hints along with input to guide a pre-trained model for solving a new task using its existing knowledge. If the hints are human-interpretable natural language (hard prompts), the related studies have been referred to as In-Context Learning [7], which enable the model to learn from task instructions, demonstrations with a few examples, or supporting information in the context. Also, the hints could be continuous vector representations (soft prompts).…”

Section: Introductionmentioning

confidence: 99%

Are Vision Transformers Robust to Patch Perturbations?

Gu¹,

Tresp²,

Qin

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Prompt engineering is a technique that involves augmenting a large pre-trained model with task-specific hints, known as prompts, to adapt the model to new tasks. Prompts can be created manually as natural language instructions or generated automatically as either natural language instructions or vector representations. Prompt engineering enables the ability to perform predictions based solely on prompts without updating model parameters, and the easier application of large pre-trained models in real-world tasks. In past years, Prompt engineering has been well-studied in natural language processing. Recently, it has also been intensively studied in vision-language modeling. However, there is currently a lack of a systematic overview of prompt engineering on pre-trained vision-language models. This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models (e.g., Flamingo), image-text matching models (e.g., CLIP), and text-to-image generation models (e.g., Stable Diffusion). For each type of model, a brief model summary, prompting methods, prompting-based applications, and the corresponding responsibility and integrity issues are summarized and discussed. Furthermore, the commonalities and differences between prompting on vision-language models, language models, and vision models are also discussed. The challenges, future directions, and research opportunities are summarized to foster future research on this topic.

show abstract

A Survey on In-context Learning

Cited by 33 publications

References 16 publications

ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction

ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction

Post Hoc Explanations of Language Models Can Improve Language Models

Are Vision Transformers Robust to Patch Perturbations?

Contact Info

Product

Resources

About