Can discrete information extraction prompts generalize across language models?

Abstract

We study whether automatically-induced prompts that effectively extract information from a language model can also be used, out-of-the-box, to probe other language models for the same information. After confirming that discrete prompts induced with the AutoPrompt algorithm outperform manual and semi-manual prompts on the slot-filling task, we demonstrate a drop in performance for Au-toPrompt prompts learned on a model and tested on another. We introduce a way to induce prompts by mixing language models at training time that results in prompts that generalize well across models. We conduct an extensive analysis of the induced prompts, finding that the more general prompts include a larger proportion of existing English words and have a less order-dependent and more uniform distribution of information across their component tokens. Our work provides preliminary evidence that it's possible to generate discrete prompts that can be induced once and used with a number of different models, and gives insights on the properties characterizing such prompts. 1

Utilizing ChatGPT to Enhance Clinical Trial Enrollment

“…LLMs have already reached state-of-the-art (SoA) performance in various tasks, and selecting an appropriate prompt has a significant impact. As a result, a significant number of related works investigate techniques to improve prompt construction further and their effectiveness [54,81,88], explore their robustness to permutations of the demonstrative examples [107], their sensitivity to negations [37], and their ability to generalize across different LLMs [79].…”

“…Rakotonirina et al [79] investigated whether prompts can generalize across different LMs and LLMs focusing on the slot filling NLP task. The authors experimented with manual, semi-manual, and automatic methods for prompt creation.…”