Entities as Experts: Sparse Memory Access with Entity Supervision

Févry, Thibault; Soares, Livio Baldini; FitzGerald, Nicholas; Choi, Eunsol; Kwiatkowski, Tom

doi:10.48550/arxiv.2004.07202

Cited by 20 publications

(37 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The entity having an alias with the smallest edit distance (Levenshtein, 1966) to the predicted text output is taken as the predicted entity. Entities as experts: Févry et al (2020) proposed EaE, a model which aims to integrate entity knowledge into a transformer-based language model. For temporal KGQA on CRONQUES-TIONS, we assume that all grounded entity and time mention spans are marked in the question 1 .…”

Section: Other Methods Comparedmentioning

confidence: 99%

“…On this new dataset, we apply approaches based on deep language models (LM) alone, such as T5 (Raffel et al, 2020), BERT (Devlin et al, 2019), and KnowBERT (Peters et al, 2019), and also hybrid LM+KG embedding approaches, such as Entities-as-Experts (Févry et al, 2020) and Em-bedKGQA (Saxena et al, 2020). We find that these baselines are not suited to temporal reasoning.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Question Answering Over Temporal Knowledge Graphs

Saxena¹,

Chakrabarti²,

Talukdar³

2021

Preprint

View full text Add to dashboard Cite

Temporal Knowledge Graphs (Temporal KGs) extend regular Knowledge Graphs by providing temporal scopes (e.g., start and end times) on each edge in the KG. While Question Answering over KG (KGQA) has received some attention from the research community, QA over Temporal KGs (Temporal KGQA) is a relatively unexplored area. Lack of broadcoverage datasets has been another factor limiting progress in this area. We address this challenge by presenting CRONQUESTIONS, the largest known Temporal KGQA dataset, clearly stratified into buckets of structural complexity. CRONQUESTIONS expands the only known previous dataset by a factor of 340×. We find that various state-of-the-art KGQA methods fall far short of the desired performance on this new dataset. In response, we also propose CRONKGQA, a transformerbased solution that exploits recent advances in Temporal KG embeddings, and achieves performance superior to all baselines, with an increase of 120% in accuracy over the next best performing method. Through extensive experiments, we give detailed insights into the workings of CRONKGQA, as well as situations where significant further improvements appear possible. In addition to the dataset, we have released our code as well.1. The underlying KG is a Temporal KG.2. The answer is either an entity or time duration. 3. Complex temporal reasoning might be needed. KG Embeddings are low-dimensional dense vector representations of entities and relations in a KG. Several methods have been proposed in the literature to embed KGs (Bordes et al. 2013, Trouillon et al. 2016, Vashishth et al. 2020). These embeddings were originally proposed for the task of KG completion i.e., predicting missing edges in the KG, since most real world KGs are incomplete. Recently, however, they have also been applied to the task of KGQA where they have been shown to increase performance the settings of both of complete and incomplete KGs (Saxena et al. 2020;).

show abstract

Section: Other Methods Comparedmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Question Answering Over Temporal Knowledge Graphs

Saxena¹,

Chakrabarti²,

Talukdar³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…KBQA A number of early approaches in ODQA focused on using structured KBs (Berant et al, 2013) such as Freebase (Bollacker et al, 2007), with recent examples from Févry et al (2020) and Verga et al (2020). This approach often has high precision but suffers when KB doesn't match user requirements, or where the schema limits what knowledge can be stored.…”

Section: Related Workmentioning

confidence: 99%

PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them

Lewis¹,

Wu²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Open-domain Question Answering models which directly leverage question-answer (QA) pairs, such as closed-book QA (CBQA) models and QA-pair retrievers, show promise in terms of speed and memory compared to conventional models which retrieve and read from text corpora. QA-pair retrievers also offer interpretable answers, a high degree of control, and are trivial to update at test time with new knowledge. However, these models lack the accuracy of retrieve-and-read systems, as substantially less knowledge is covered by the available QA-pairs relative to text corpora like Wikipedia. To facilitate improved QA-pair models, we introduce Probably Asked Questions (PAQ), a very large resource of 65M automatically-generated QA-pairs. We introduce a new QA-pair retriever, RePAQ, to complement PAQ. We find that PAQ preempts and caches test questions, enabling RePAQ to match the accuracy of recent retrieve-and-read models, whilst being significantly faster. Using PAQ, we train CBQA models which outperform comparable baselines by 5%, but trail RePAQ by over 15%, indicating the effectiveness of explicit retrieval. RePAQ can be configured for size (under 500MB) or speed (over 1K questions per second) whilst retaining high accuracy. Lastly, we demonstrate RePAQ's strength at selective QA, abstaining from answering when it is likely to be incorrect. This enables RePAQ to "back-off" to a more expensive state-of-the-art model, leading to a combined system which is both more accurate and 2x faster than the state-of-the-art model alone.

show abstract

“…Use of knowledge augmented neural networks had been explored in pre-Transformer era as well (Weston et al, 2014;Sukhbaatar et al, 2015). More recently, in the context of Transformers, Févry et al (2020) utilized an explicit key-value memory to store entity representations, which are trained along with the rest of model in an end-to-end manner. Verga et al (2020) build on Févry et al (2020), and introduced Facts as Expert (FaE) model with explicit symbolic memory of (subject, relation, object) triples based on end-to-end trained entity representations.…”

Section: Related Workmentioning

confidence: 99%

“…More recently, in the context of Transformers, Févry et al (2020) utilized an explicit key-value memory to store entity representations, which are trained along with the rest of model in an end-to-end manner. Verga et al (2020) build on Févry et al (2020), and introduced Facts as Expert (FaE) model with explicit symbolic memory of (subject, relation, object) triples based on end-to-end trained entity representations. Notably, one of the motivations behind FaE is the ease of updating knowledge by directly modifying the content of the explicit symbolic memory.…”

Section: Related Workmentioning

confidence: 99%

Modifying Memories in Transformer Models

Zhu,

Rawat,

Zaheer

et al. 2020

Preprint

View full text Add to dashboard Cite

Large Transformer models have achieved impressive performance in many natural language tasks. In particular, Transformer based language models have been shown to have great capabilities in encoding factual knowledge in their vast amount of parameters. While the tasks of improving the memorization and generalization of Transformers have been widely studied, it is not well known how to make transformers forget specific old facts and memorize new ones. In this paper, we propose a new task of explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts. This task is useful in many scenarios, such as updating stale knowledge, protecting privacy, and eliminating unintended biases stored in the models. We benchmarked several approaches that provide natural baseline performances on this task. This leads to the discovery of key components of a Transformer model that are especially effective for knowledge modifications. The work also provides insights into the role that different training phases (such as pretraining and fine-tuning) play towards memorization and knowledge modification.

show abstract

Entities as Experts: Sparse Memory Access with Entity Supervision

Cited by 20 publications

References 0 publications

Question Answering Over Temporal Knowledge Graphs

Question Answering Over Temporal Knowledge Graphs

PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them

Modifying Memories in Transformer Models

Contact Info

Product

Resources

About