Improving and Simplifying Pattern Exploiting Training

Tam, Derek; Menon, Rakesh R.; Bansal, Mohit; Srivastava, Shashank; Raffel, Colin

doi:10.18653/v1/2021.emnlp-main.407

Cited by 48 publications

(36 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These prompts share the same format of masked language modeling, the pre-training tasks of many pre-trained LMs, and thus leads to improved few-shot performance. Extending from PET, Gao et al (2020) proposed LM-BFF which learns to generate prompts automatically and incorporates demonstrations into the input; Tam et al (2021) proposed ADAPET which densifies the supervision signal with a label conditioning objective.…”

Section: Related Workmentioning

confidence: 99%

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

Ye¹,

Lin²,

Ren³

2021

Preprint

View full text Add to dashboard Cite

Humans can learn a new language task more efficiently than machines, conceivably by leveraging their prior experience and knowledge in learning other tasks. In this paper, we explore whether such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CROSSFIT, a task setup for studying cross-task few-shot learning ability, which standardizes seen/unseen task splits, data access during different learning stages, and the evaluation protocols. In addition, we present NLP Few-shot Gym, a repository of 160 few-shot NLP tasks, covering diverse task categories and applications, and converted to a unified text-to-text format.Our empirical analysis reveals that the fewshot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. Additionally, the advantage lasts into medium-resource scenarios when thousands of training examples are available. We also observe that selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability. 1

show abstract

Section: Related Workmentioning

confidence: 99%

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

Ye¹,

Lin²,

Ren³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…A new fine-tuning methodology named prompt-tuning has arisen: adapting the pre-trained language model directly as a predictor through completion of a cloze task. Prompt-tuning for pre-trained language models is a rapidly emerging field in natural language processing [40,46,71] and have attracted lots of attention. Originally from GPT-3, prompt-tuning has been applied to various of tasks including relation extraction [20], event extraction [21,59], named entity recognition [5,7], entity typing [13], and so on.…”

Section: Prompt-tuningmentioning

confidence: 99%

Ontology-enhanced Prompt-tuning for Few-shot Learning

Ye,

Zhang,

Deng

et al. 2022

Preprint

View full text Add to dashboard Cite

Few-shot Learning (FSL) is aimed to make predictions based on a limited number of samples. Structured data such as knowledge graphs and ontology libraries has been leveraged to benefit the few-shot setting in various tasks. However, the priors adopted by the existing methods suffer from challenging knowledge missing, knowledge noise, and knowledge heterogeneity, which hinder the performance for few-shot learning. In this study, we explore knowledge injection for FSL with pre-trained language models and propose ontology-enhanced prompt-tuning (OntoPrompt). Specifically, we develop the ontology transformation based on the external knowledge graph to address the knowledge missing issue, which fulfills and converts structure knowledge to text. We further introduce span-sensitive knowledge injection via a visible matrix to select informative knowledge to handle the knowledge noise issue. To bridge the gap between knowledge and text, we propose a collective training algorithm to optimize representations jointly. We evaluate our proposed OntoPrompt in three tasks, including relation extraction, event extraction, and knowledge graph completion, with eight datasets. Experimental results demonstrate that our approach can obtain better few-shot performance than baselines.

show abstract

“…While PET requires task-specific prompts, it achieves better performance than GPT-3 in-context with smaller models [26]. ADAPET improves upon PET by providing more supervision during fine-tuning [27]. LM-BFF [11] improves prompt-based fine-tuning by dynamically constructing prompts.…”

Section: Few-shot Learning In Nlpmentioning

confidence: 99%

RAFT: A Real-World Few-Shot Text Classification Benchmark

Alex¹,

Lifland²,

Tunstall³

et al. 2021

Preprint

View full text Add to dashboard Cite

Large pre-trained language models have shown promise for few-shot learning, completing text-based tasks given only a few task-specific examples. Will models soon solve classification tasks that have so far been reserved for human research assistants? Existing benchmarks are not designed to measure progress in applied settings, and so don't directly answer this question. The RAFT benchmark (Realworld Annotated Few-shot Tasks) focuses on naturally occurring tasks and uses an evaluation setup that mirrors deployment. Baseline evaluations on RAFT reveal areas current techniques struggle with: reasoning over long texts and tasks with many classes. Human baselines show that some classification tasks are difficult for non-expert humans, reflecting that real-world value sometimes depends on domain expertise. Yet even non-expert human baseline F1 scores exceed GPT-3 by an average of 0.11. The RAFT datasets and leaderboard will track which model improvements translate into real-world benefits at https://raft.elicit.org.

show abstract

Improving and Simplifying Pattern Exploiting Training

Cited by 48 publications

References 11 publications

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

Ontology-enhanced Prompt-tuning for Few-shot Learning

RAFT: A Real-World Few-Shot Text Classification Benchmark

Contact Info

Product

Resources

About