PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts

Bach, Stephen; Sanh, Victor; Yong, Zheng; Webson, Albert; Raffel, Colin; Nayak, Nihal V.; Sharma, Abheesht; Kim, Tae-Woon; Saiful, Bari, M; Févry, Thibault; Alyafeai, Zaid; Dey, Manan; Santilli, Andrea; Sun, Zhiqing; Ben-david, Srulik; Xu, Canwen; Chhablani, Gunjan; Wang, Han; Fries, Jason; Al-shaibani, Maged; Sharma, Shanya; Thakker, Urmish; Almubarak, Khalid; Tang, Xiangru; Radev, Dragomir; Jiang, Mike Tian-jian; Rush, Alexander M.

doi:10.18653/v1/2022.acl-demo.9

Cited by 44 publications

(42 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…verbalizers can have a significant effect on final performance (Bach et al, 2022). Here we choose three alternative templates from PromptSource 5 for the NLI task, as follows:…”

Section: Additional Studiesmentioning

confidence: 99%

“…Prompt optimization and ensembles in language models. Previous work has shown that the prompt order (Lu et al, 2021), how each task is verbalized (Bach et al, 2022), and the distribution of labels in the prompts (Zhao et al, 2021) can all affect final task performance. In this paper, we find that, when shifting from the paradigm of (input → output) pairs to (input, rationale → output) pairs, there is also a large variance in the final task performance when the rationales used in the prompts differ.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Rationale-Augmented Ensembles in Language Models

Wang¹,

Lee²,

Schuurmans³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent research has shown that rationales, or step-by-step chains of thought, can be used to improve performance in multi-step reasoning tasks. We reconsider rationale-augmented prompting for few-shot in-context learning, where (input → output) prompts are expanded to (input, rationale → output) prompts. For rationale-augmented prompting we demonstrate how existing approaches, which rely on manual prompt engineering, are subject to sub-optimal rationales that may harm performance. To mitigate this brittleness, we propose a unified framework of rationale-augmented ensembles, where we identify rationale sampling in the output space as the key component to robustly improve performance. This framework is general and can easily be extended to common natural language processing tasks, even those that do not traditionally leverage intermediate steps, such as question answering, word sense disambiguation, and sentiment analysis. We demonstrate that rationale-augmented ensembles achieve more accurate and interpretable results than existing prompting approaches-including standard prompting without rationales and rationale-based chain-of-thought prompting-while simultaneously improving interpretability of model predictions through the associated rationales.Preprint.

show abstract

“…verbalizers can have a significant effect on final performance (Bach et al, 2022). Here we choose three alternative templates from PromptSource 5 for the NLI task, as follows:…”

Section: Additional Studiesmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Rationale-Augmented Ensembles in Language Models

Wang¹,

Lee²,

Schuurmans³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Figure 15 shows an example of applying prompt through Promptsource (Bach et al, 2022) as mentioned in Section 3.1.…”

Section: E Examples Of Applying Promptsmentioning

confidence: 99%

“…: Known Unknown, Logic Grid, StrategyQA, Hindu Knowledge, Movie Dialog, Code Description, Conceptual, Language ID, Vitamin C, Syllogisms, Misconceptions, Logical Deduction, Winowhy and Novel Concepts.Figure 15: Example of applying prompt to a given instance through Promptsource(Bach et al, 2022).…”

mentioning

confidence: 99%

See 1 more Smart Citation

Retrieval of Soft Prompt Enhances Zero-Shot Task Generalization

Ye¹,

Jang²,

Kim³

et al. 2022

Preprint

View full text Add to dashboard Cite

During zero-shot inference with language models (LMs), using hard prompts alone may not be able to fully describe the target task. In this paper, we explore how the retrieval of soft prompts obtained through prompt tuning can assist hard prompts in zero-shot task generalization. Specifically, we train soft prompt embeddings for each prompt through prompt tuning, store the samples of the training instances (hard prompt + input instances) mapped with the prompt embeddings, and retrieve the corresponding prompt embedding of the training instance closest to the query instance during inference. Results show this simple approach enhances the performance of T0 on unseen tasks by outperforming it on 10 out of 11 datasets as well as improving the mean accuracy of T0 on BIG-bench benchmark by 2.39% points while adding only 0.007% additional parameters. Also, using interpolation of multiple embeddings and variance-based ranking further improve accuracy and robustness to different evaluation prompts, widening the performance gap. Finally, we find that retrieving source embeddings trained on similar answer choice formats is more important than those on similar task types. 1

show abstract