LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

Abstract

Entity representations are useful in natural language tasks involving entities. In this paper, we propose new pretrained contextualized representations of words and entities based on the bidirectional transformer (Vaswani et al., 2017). The proposed model treats words and entities in a given text as independent tokens, and outputs contextualized representations of them. Our model is trained using a new pretraining task based on the masked language model of BERT (Devlin et al., 2019). The task involves predicting randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia. We also propose an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the transformer, and considers the types of tokens (words or entities) when computing attention scores. The proposed model achieves impressive empirical performance on a wide range of entity-related tasks. In particular, it obtains state-of-the-art results on five well-known datasets: Open Entity (entity typing), TACRED (relation classification), CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), and SQuAD 1.1 (extractive question answering). Our source code and pretrained representations are available at https: //github.com/studio-ousia/luke. IntroductionMany natural language tasks involve entities, e.g., relation classification, entity typing, named entity recognition (NER), and question answering (QA). Key to solving such entity-related tasks is a model to learn the effective representations of entities. Conventional entity representations assign each entity a fixed embedding vector that stores information regarding the entity in a knowledge base (KB)

GEMNET: Effective Gated Gazetteer Representations for Recognizing Complex Entities in Low-context Input

“…There are also limitations in the integration of gazetteer features. Existing studies often add extra features to a word-level model's Contextual Word Representations (CWRs), which typically contain no info about real world entities or their spans (Yamada et al, 2020). This concatenation approach is sub-optimal as it creates additional, and often highly correlated features.…”

“…While a subtagger may learn regularities in entity names, a key limitation is that it needs retraining and evaluation on gazetteer updates. Recent work has considered directly integrating knowledge into transformers, e.g., KnowBert adds knowledge to BERT layers (Peters et al, 2019), and LUKE is pretrained to predict masked entities (Yamada et al, 2020). The drawbacks of such methods are that they are specific to Transformers, and the model's knowledge cannot be updated without retraining.…”

Automated Concatenation of Embeddings for Structured Prediction

“…In NER, we also conduct a comparison on the revised version of German datasets in the CoNLL 2006 shared task (Buchholz and Marsi, 2006). Recent work such as Yu et al (2020) and Yamada et al (2020) utilizes document contexts in the datasets. We follow their work and extract document embeddings for the transformer-based embeddings.…”

Entity Resolution in Open-domain Conversations

“…NEL typically involves two tasks: recognizing named entities in a given text and then disamgibuating the entity mentions according to the knowledge base (KB). Researchers have shown great success in NER with the help of Convolutional Neural Networks (CNNs), Bidirectional Recurrent Neural Networks (Bi-RNNs), and attention mechanisms along with a CRF decoder (Chiu and Nichols, 2016;Akbik et al, 2018;Ghaddar and Langlais, 2018;Jiang et al, 2019;Baevski et al, 2019;Yamada et al, 2020). Deep neural networks (DNNs) are also dominant in entity resolution tasks.…”