Matching the Blanks: Distributional Similarity for Relation Learning

Abstract

General purpose relation extractors, which can model arbitrary relations, are a core aspiration in information extraction. Efforts have been made to build general purpose extractors that represent relations with their surface forms, or which jointly embed surface forms with relations from an existing knowledge graph. However, both of these approaches are limited in their ability to generalize. In this paper, we build on extensions of Harris' distributional hypothesis to relations, as well as recent advances in learning text representations (specifically, BERT), to build task agnostic relation representations solely from entity-linked text. We show that these representations significantly outperform previous work on exemplar based relation extraction (FewRel) even without using any of that task's training data. We also show that models initialized with our task agnostic representations, and then tuned on supervised relation extraction datasets, significantly outperform the previous methods on Se-mEval 2010 Task 8, KBP37, and TACRED. * Work done as part of the Google AI residency.

Learning from Context or Names? An Empirical Study on Neural Relation Extraction

“…BERT BERT is a pre-trained language model that has been widely used in NLP tasks. We use BERT for RE following Baldini Soares et al (2019). In short, we highlight entity mentions in sentences by special markers and use the concatenations of entity representations for classification.…”

“…The following two formats are adopted by previous literature and are close to the real-world RE scenarios: (with both context and highlighted entity mentions) is provided. To let the models know where the entity mentions are, we use position embeddings (Zeng et al, 2014) for the CNN model and special entity markers (Zhang et al, 2019;Baldini Soares et al, 2019) for the pre-trained BERT. Context+Type (C+T) We replace entity mentions with their types provided in TACRED.…”

“…engines (Xiong et al, 2017). With the recent advance of deep learning, neural relation extraction (NRE) models (Socher et al, 2012;Liu et al, 2013;Baldini Soares et al, 2019) have achieved the latest state-of-the-art results and some of them are even comparable with human performance on several public RE benchmarks.…”

Winners at W-NUT 2020 Shared Task-3: Leveraging Event Specific and Chunk Span information for Extracting COVID Entities from Tweets

“…A variety of slot-filling approaches have been built on top of these deep learning advancements (Kurata et al, 2016;Qin et al, 2019). The proposed baseline for our task (Zong et al, 2020) modifies Bert model for slot-filling problem inspired by Baldini Soares et al (2019). Due to the excellent performance offered by Bert (Devlin et al, 2019) and Baldini Soares et al 2019, we build upon this baseline approach.…”

“…It takes a tweet t as input and encloses the candidate slot s, within the tweet, inside special entity start < E > and end < /E > markers. The Bert hidden representation of token < E > is then processed through a fully connected layer with softmax activation to make the binary prediction for a task (Baldini Soares et al, 2019). Since many slot-filling Tweet CT-Bert [CLS] ... Figure 2: Slot-Filling Model, described in Section §4.1.…”