Neural Snowball for Few-Shot Relation Learning

Gao, Tianyu; Han, Xu; Xie, Rong; Liu, Zhiyuan; Lin, Feng; Lin, Leyu; Sun, Maosong

doi:10.1609/aaai.v34i05.6281

Cited by 51 publications

(26 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Meta-learning based methods [11,39,40] have shown tremendous success for few-shot learning in many tasks such as few-shot image generation [47], image classification [56], and domain adaptation [63]. Following the success of such approaches, few-shot learning in NLP have been investigated for tasks such as text classification [14,59,67], entity-relation extraction [13,34], and few-shot slot filling [12,21,33]. The authors in [33] exploited regular expressions for few-shot slot filling, Prototypical Network was employed in [12], and the authors in [21] extended the CRF model by introducing collapsed dependency transition to transfer label dependency patterns.…”

Section: Related Workmentioning

confidence: 99%

Linguistically-Enriched and Context-AwareZero-shot Slot Filling

Siddique

Jamour

Hristidis

2021

Proceedings of the Web Conference 2021

View full text Add to dashboard Cite

Slot filling is identifying contiguous spans of words in an utterance that correspond to certain parameters (i.e., slots) of a user request/query. Slot filling is one of the most important challenges in modern task-oriented dialog systems. Supervised approaches have proven effective at tackling this challenge, but they need a significant amount of labeled training data in a given domain. However, new domains (i.e., unseen in training) may emerge after deployment. Thus, it is imperative that these models seamlessly adapt and fill slots from both seen and unseen domains -unseen domains contain unseen slot types with no training data, and even seen slots in unseen domains are typically presented in different contexts. This setting is commonly referred to as zero-shot slot filling. Little work has focused on this setting, with limited experimental evaluation. Existing models that mainly rely on contextindependent embedding-based similarity measures fail to detect slot values in unseen domains or do so only partially. We propose a new zero-shot slot filling neural model, LEONA, which works in three steps.Step one acquires domain-oblivious, context-aware representations of utterance words by exploiting (a) linguistic features such as part-of-speech tags; (b) named entity recognition cues; and (c) contextual embeddings from pre-trained language models.Step two fine-tunes these rich representations and produces slotindependent tags for each word.Step three exploits generalizable context-aware utterance-slot similarity features at the word level, uses slot-independent tags, and contextualizes them to produce slot-specific predictions for each word. Our thorough evaluation on four diverse public datasets demonstrates that our approach consistently outperforms state-of-the-art models by 17.52%, 22.15%, 17.42%, and 17.95% on average for unseen domains on SNIPS, ATIS, MultiWOZ, and SGD datasets, respectively.

show abstract

Section: Related Workmentioning

confidence: 99%

Linguistically-Enriched and Context-AwareZero-shot Slot Filling

Siddique

Jamour

Hristidis

2021

Proceedings of the Web Conference 2021

View full text Add to dashboard Cite

show abstract

“…Due to the scarcity of labeled training data, FL faces the problem of over-fitting. Existing methods to overcome overfitting include: (i) model-based methods that explore how to reduce the hypothesis space of the fewshot task (Triantafillou et al, 2017;Hu et al, 2018), (ii) data-based methods that try to augment additional data to the few-shot set (Benaim & Wolf, 2018;Gao et al, 2020b), and (iii) algorithmbased solutions that aim to improve strategies for searching for the best hypothesis. Recently, a new paradigm introducing prompts achieves promising results for few-shot language learning as shown by GPT-3 (Brown et al, 2020), PET (Schick & Schütze, 2020) and LM-BFF (Gao et al, 2020a).…”

Section: Few-shot Learningmentioning

confidence: 99%

LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5

Qin¹,

Joty²

2021

Preprint

View full text Add to dashboard Cite

Existing approaches to lifelong language learning rely on plenty of labeled data for learning a new task, which is hard to obtain in most real scenarios. Considering that humans can continually learn new tasks from a handful of examples, we expect the models also to be able to generalize well on new few-shot tasks without forgetting the previous ones. In this work, we define this more challenging yet practical problem as Lifelong Few-shot Language Learning (LFLL) and propose a unified framework for it based on prompt tuning of T5. Our framework called LFPT5 takes full advantage of PT's strong few-shot learning ability, and simultaneously trains the model as a task solver and a data generator. Before learning a new domain of the same task type, LFPT5 generates pseudo (labeled) samples of previously learned domains, and later gets trained on those samples to alleviate forgetting of previous knowledge as it learns the new domain. In addition, a KL divergence loss is minimized to achieve label consistency between the previous and the current model. While adapting to a new task type, LFPT5 includes and tunes additional prompt embeddings for the new task. With extensive experiments, we demonstrate that LFPT5 can be applied to various different types of tasks and significantly outperform previous methods in different LFLL settings.

show abstract

“…The ProtoNet [199] is popular in metric-based LSL RE/C tasks [56,72,252] including designs multi-level attention schemes to highlight the crucial instances and features respectively [72], adopting the large-margin ProtoNet with fine-grained sentence features [56], integrating external descriptions to enhance the origin ProtoNet [247], and using a multi-level structure to encode the metric prototype through various stage [252]. Metric-based LSL has also been integrated with semi-supervised learning, exploring novel relations from the seed relation set with a relational metric and snowball strategy [73]. IV.…”

Section: IIImentioning

confidence: 99%

What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data

Hu¹,

Chapman²,

Wen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Supervised machine learning has several drawbacks that make it difficult to use in many situations. Drawbacks include: heavy reliance on massive training data, limited generalizability and poor expressiveness of high-level semantics. Low-shot Learning attempts to address these drawbacks. Low-shot learning allows the model to obtain good predictive power with very little or no training data, where structured knowledge plays a key role as a high-level semantic representation of human. This article will review the fundamental factors of low-shot learning technologies, with a focus on the operation of structured knowledge under different low-shot conditions. We also introduce other techniques relevant to low-shot learning. Finally, we point out the limitations of low-shot learning, the prospects and gaps of industrial applications, and future research directions.

show abstract

Neural Snowball for Few-Shot Relation Learning

Cited by 51 publications

References 18 publications

Linguistically-Enriched and Context-AwareZero-shot Slot Filling

Linguistically-Enriched and Context-AwareZero-shot Slot Filling

LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5

What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data

Contact Info

Product

Resources

About