Entities as Experts: Sparse Memory Access with Entity Supervision

Abstract

We focus on the problem of capturing declarative knowledge about entities in the learned parameters of a language model. We introduce a new model-Entities as Experts (EAE)that can access distinct memories of the entities mentioned in a piece of text. Unlike previous efforts to integrate entity knowledge into sequence models, EAE's entity representations are learned directly from text. We show that EAE's learned representations capture sufficient knowledge to answer TriviaQA questions such as "Which Dr. Who villain has been played by Roger Delgado, Anthony Ainley, Eric Roberts?", outperforming an encodergenerator Transformer model with 10× the parameters. According to the LAMA knowledge probes, EAE contains more factual knowledge than a similarly sized BERT, as well as previous approaches that integrate external sources of entity knowledge. Because EAE associates parameters with specific entities, it only needs to access a fraction of its parameters at inference time, and we show that the correct identification and representation of entities is essential to EAE's performance. * Work done during Google AI residency. † Work done at Google Research.

Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training

“…Incorporating KGs Most prior works on incorporating KG with text often learn KG entity representations and add them to the mention spans linked to the entity Févry et al, 2020) or create subgraphs relevant to the query that are expanded with text in the embedding space Sun et al, 2019;Xiong et al, 2019). Some others incorporate additional modules.…”

“…Some others incorporate additional modules. Verga et al (2020) extend Févry et al (2020) by adding a triple memory with (subject, relation) encoding as the key and the object encoding as the value. Das et al (2017) use universal schema (Riedel et al, 2013) that embeds text and KGs in a shared space for their integration.…”

Large Product Key Memory for Pretrained Language Models

“…In addition to , augmenting memory architecture to a language model is a promising research direction. For example, EaE (Févry et al, 2020) and FaE (Verga et al, 2020) jointly train a memory that is interleaved in a transformer and dedicated to entities (or facts) with sparse updates, and access to only a small portion of the memory in inference time. On the other hand, each memory slot in and ours does not have explicit meaning.…”

Entity Linking in 100 Languages

“…Entity linking (EL) fulfils a key role in grounded language understanding: Given an ungrounded entity mention in text, the task is to identify the entity's corresponding entry in a Knowledge Base (KB). In particular, EL provides grounding for applications like Question Answering (Févry et al, 2020b) (also via Semantic Parsing (Shaw et al, 2019)) and Text Generation (Puduppully et al, 2019); it is also an essential component in knowledge base population (Shen et al, 2014). Entities have played a growing role in representation learning.…”