Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP

Abstract

Retrieval is a core component for open-domain NLP tasks. In open-domain tasks, multiple entities can share a name, making disambiguation an inherent yet under-explored problem. We propose an evaluation benchmark for assessing the entity disambiguation capabilities of these retrievers, which we call Ambiguous Entity Retrieval (AmbER) sets. We define an AmbER set as a collection of entities that share a name along with queries about those entities. By covering the set of entities for polysemous names, AmbER sets act as a challenging test of entity disambiguation. We create AmbER sets for three popular open-domain tasks: fact checking, slot filling, and question answering, and evaluate a diverse set of retrievers. We find that the retrievers exhibit popularity bias, significantly under-performing on rarer entities that share a name, e.g., they are twice as likely to retrieve erroneous documents on queries for the less popular entity under the same name. These experiments on AmbER sets show their utility as an evaluation tool and highlight the weaknesses of popular retrieval systems. 1

TABi: Type-Aware Bi-Encoders for Open-Domain Entity Retrieval

“…L type (Q) uses type labels to form positive and negative pairs over queries. 6 Let P type (q) be the set of all queries in a batch that share the same type t as a query q and N type (q) be the other queries in the batch with a different type. Then L type (Q) is:…”

“…Retrieving the correct George Washington in the query above-George Washington the baseball player, rather than George Washington the president-requires the retriever to recognize that keywords "team" and "play" imply George Washington is an athlete. However, recent work has shown that state-ofthe-art retrievers exhibit popularity biases and struggle to resolve ambiguous mentions of rare "tail" entities [6].…”

Entity-Based Knowledge Conflicts in Question Answering

“…The framework maps a QA instance x = (q, a, c), with query q, answer a, and the context passage c in which a appears, to x = (q, a , c ) where a is replaced by substitution answer a as the gold answer, and where all occurrences of a in c have been replaced with a , producing new context c . This substitution framework extends partiallyautomated dataset creation techniques introduced by Chen et al (2021) for Ambiguous Entity Retrieval (AmbER). Our dataset derivation follows two steps: (1) identifying QA instances with named entity answers, and (2) replacing all occurrences of the answer in the context with a substituted entity, effectively changing the answer.…”

“…How does Popularity of an Answer Entity impact Memorization? Using popularity substitution we examine if models are biased towards predicting more popular answers (Shwartz et al, 2020;Chen et al, 2021). Limiting our focus to the Person answer category, we order all PER Wikidata entities by popularity (approximated by Wikipedia monthly page views) and stratify them into five evenly sized popularity buckets.…”

Namesakes: Ambiguously Named Entities from Wikipedia and News

“…WikiDisamb30 (Ferragina and Scaiella, 2012), ACE and MSNBC (Ratinov et al, 2011), WNED-CWEB and WNED-WIKI (Guo and Barbosa, 2018) CoNLL-YAGO (Hoffart et al, 2011), and TAC KBP Entity Discovery and Linking dataset (Ji et al, 2017). The recently introduced Ambiguous Entity Retrieval (AmbER) dataset by Chen et al (2021) is an exception, including subsets of identically named entities for the purpose of fact checking, slot filling, and question-answering tasks. AmBer is limited to Wikipedia text and was automatically generated.…”