Reframing Instructional Prompts to GPTk’s Language

Mishra, Swaroop; Khashabi, Daniel; Baral, Chitta; Choi, Yejin; Hajishirzi, Hannaneh

doi:10.18653/v1/2022.findings-acl.50

Cited by 47 publications

(53 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FLAN and T0 (Sanh et al, 2021) of tasks and achieving zero-shot generalization on target unseen tasks. Task reframing (Mishra et al, 2021a) proposed several guidelines to reframe task instructions to improve model response to follow instructions. Analysis introduced to understand in-context learning better on a large set of training tasks (Min et al, 2021(Min et al, , 2022.…”

Section: Related Workmentioning

confidence: 99%

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

Parmar¹,

Mishra²,

Purohit³

et al. 2022

Findings of the Association for Computational Linguistics: NAACL 2022

Self Cite

View full text Add to dashboard Cite

Single-task models have proven pivotal in solving specific tasks; however, they have limitations in real-world applications where multitasking is necessary and domain shifts are exhibited. Recently, instructional prompts have shown significant improvement towards multitask generalization; however, the effect of instructional prompts and Multi-Task Learning (MTL) has not been systematically studied in the biomedical domain. Motivated by this, this paper explores the impact of instructional prompts for biomedical MTL. We introduce the BoX, a collection of 32 instruction tasks for Biomedical NLP across (X) various categories. Using this meta-dataset, we propose a unified model termed as In-BoXBART, that can jointly learn all tasks of the BoX without any task-specific modules. To the best of our knowledge, this is the first attempt to propose a unified model in the biomedical domain and use instructions to achieve generalization across several biomedical tasks. Experimental results indicate that the proposed model: 1) outperforms single-task baseline by ∼3% and multitask (without instruction) baseline by ∼18% on an average, and 2) shows ∼23% improvement compared to single-task baseline in few-shot learning (i.e., 32 instances per task) on an average. Our analysis indicates that there is significant room for improvement across tasks in the BoX, implying the scope for future research direction. 1

show abstract

Section: Related Workmentioning

confidence: 99%

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

Parmar¹,

Mishra²,

Purohit³

et al. 2022

Findings of the Association for Computational Linguistics: NAACL 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…The release of GPT-3 (Brown et al, 2020) initiated a body of work on the emergent ability of LMs to follow discrete natural language prompts. Consequently, several followup studies have used manually-designed discrete prompts for probing LMs (Petroni et al, 2019;Jiang et al, 2020), improving LMs' few-shot ability (Schick and Schütze, 2021;Gao et al, 2021;Le Scao and Rush, 2021), and their zero-shot ability as well as transferability (Mishra et al, 2022a;Reynolds and McDonell, 2021). Most importantly, discrete prompts have the advantages of being human-readable and thus easily interpretable though we do not have efficient and algorithmic ways of reconstructing them.…”

Section: Related Workmentioning

confidence: 99%

“…For example, Shin et al (2020)'s algorithm discovers discrete prompts, yet the results are not human readable. Prior work also finds that model performance is highly sensitive to small changes in wordings (Mishra et al, 2022a) and that optimization over the discrete prompt space is non-trivial and often highly unstable. Our findings here about the disconnect between continuous prompts and their discrete interpretation provides another perspective on the difficulty of discovering discrete prompts via continuous optimization algorithms that (directly or indirectly) leverage the continuous space (more discussion in §6).…”

Section: Related Workmentioning

confidence: 99%

Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts

Khashabi¹,

Lyu²,

Min³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning. Motivated by these promising results, we investigate the feasibility of extracting a discrete (textual) interpretation of continuous prompts that is faithful to the problem they solve. In practice, we observe a "wayward" behavior between the task solved by continuous prompts and the nearest neighbor discrete projections of these prompts: One can find continuous prompts that solve a task while being projected to an arbitrary text (e.g., definition of a different or even a contradictory task) and simultaneously being within a very small (2%) margin of the best continuous prompt of the same size for the task. We provide intuitions behind this odd and surprising behavior, as well as extensive empirical analyses quantifying the effect of design choices. For instance, larger models exhibit higher waywardness, i.e, we can find prompts that more closely map to any arbitrary text with a smaller drop of accuracy. These findings have important implications relating to the difficulty of faithfully interpreting continuous prompts and their generalization across models and tasks, providing guidance for future progress in prompting language models.

show abstract

“…PROMPTING) proposed by GPT (Radford et al, 2019;Brown et al, 2020), which directly do prediction via PLM without any training. With well-designed prompts, large-scale PLMs have shown its notable zero-shot learning ability in various natural language processing tasks (Jiang et al, 2020a;Shin et al, 2020;Reynolds and Mc-Donell, 2021;Mishra et al, 2021).…”

Section: Zero-shot Learning With Plmmentioning

confidence: 99%

ZeroGen$^+$: Self-Guided High-Quality Data Generation in Efficient Zero-Shot Learning

Gao¹,

Pi²,

Ye³

et al. 2022

Preprint

View full text Add to dashboard Cite

Nowadays, owing to the superior capacity of the large pre-trained language models (PLM), the PLM-based zero-shot learning has shown promising performances on various natural language processing tasks. There are emerging interests in further exploring the zero-shot learning potential of PLMs. Among them, Ze-roGen (Ye et al., 2022a) attempts to purely use PLM to generate data and train a tiny model without relying on any task-specific annotation. Despite its remarkable results, we observe that the synthesized data from PLM contains a significant portion of samples with low quality, overfitting on such data greatly hampers the performance of the trained model and makes it unreliable for deployment. Since no gold data is accessible in zero-shot scenario, it is hard to perform model/data selection to prevent overfitting to the low-quality data. To address this problem, we propose a noise-robust bi-level re-weighting framework which is able to learn the per-sample weights measuring the data quality without requiring any gold data. With the learnt weights, clean subsets of different sizes can then be sampled to train the task model. We theoretically and empirically verify our method is able to construct synthetic dataset with good quality. Our method yeilds a 7.1% relative improvement than ZEROGEN on average accuracy across five different established text classification tasks.

show abstract

Reframing Instructional Prompts to GPTk’s Language

Cited by 47 publications

References 18 publications

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts

ZeroGen$^+$: Self-Guided High-Quality Data Generation in Efficient Zero-Shot Learning

Contact Info

Product

Resources

About