FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation

Abstract

We present a Few-Shot Relation Classification Dataset (FewRel), consisting of 70, 000 sentences on 100 relations derived from Wikipedia and annotated by crowdworkers. The relation of each sentence is first recognized by distant supervision methods, and then filtered by crowdworkers. We adapt the most recent state-of-the-art few-shot learning methods for relation classification and conduct thorough evaluation of these methods. Empirical results show that even the most competitive few-shot learning models struggle on this task, especially as compared with humans. We also show that a range of different reasoning skills are needed to solve our task. These results indicate that few-shot relation classification remains an open problem and still requires further research. Our detailed analysis points multiple directions for future research. All details and resources about the dataset and baselines are released on

Taxonomical hierarchy of canonicalized relations from multiple Knowledge Bases

“…Although Freebase has more than 700 properties, only 52 could qualify as relation because of the underlying corpus. Recently published datasets TACRED (Zhang et al, 2017) and FewRel (Han et al, 2018b), cover 42 and 100 relations respectively. Similar to our work, TACRED considers relations specific to person, organization and location entity types.…”

“…The first limitation is because of the pre-defined handcrafted or corpus-dependent relations list (Mitchell et al, 2005;Hendrickx et al, 2009). To scale the number of relations a few datasets (Riedel et al, 2010;Han et al, 2018b) use a single KB to obtain a potential list of relations. As there is no standard nomenclature and mapping followed among KBs, restriction to a single KB leads to the second bottleneck.…”

OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction

“…Although the current NRE models are effective and have been applied for various scenarios, including supervised learning paradigm (Zeng et al, 2014a;Nguyen and Grishman, 2015;Zhang et al, 2015;Zhou et al, 2016), distantly supervised learning paradigm (Zeng et al, 2015;Lin et al, 2016;Han et al, 2018b), few-shot learning paradigm (Han et al, 2018c;Gao et al, 2019;Ye and Ling, 2019;Soares et al, 2019;Zhang et al, 2019), there still lack an effective and stable toolkit to support the implementation, deployment and evaluation of models. In fact, for other tasks related to RE, there have been already some effective and long-term maintained toolkits, such as Spacy 1 for named entity recognition (NER), TagMe (Ferragina and Scaiella, 2010) for entity linking (EL), OpenKE (Han et al, 2018a) for knowledge embedding, and Stanford OpenIE (Angeli et al, 2015) for open information extraction.…”

Adapting Meta Knowledge Graph Information for Multi-Hop Reasoning over Few-Shot Relations

“…Most previous work assumes that there are enough triples to train an effective and robust reasoning models for each relation in KGs. However, as shown in Figure 1, a large portion of KG relations are actually long-tail Han et al, 2018) and only contain few triples, which can be called few-shot relations. Some pilot experiments show that the performance of previous multi-hop reasoning models, e.g., MIN-ERVA (Das et al, 2018) and MultiHop (Lin et al, 2018), on these few-shot relations will drop significantly.…”