What do Entity-Centric Models Learn? Insights from Entity Linking in Multi-Party Dialogue

Abstract

Humans use language to refer to entities in the external world. Motivated by this, in recent years several models that incorporate a bias towards learning entity representations have been proposed. Such entity-centric models have shown empirical success, but we still know little about why.In this paper we analyze the behavior of two recently proposed entity-centric models in a referential task, Entity Linking in Multi-party Dialogue (SemEval 2018 Task 4). We show that these models outperform the state of the art on this task, and that they do better on lower frequency entities than a counterpart model that is not entity-centric, with the same model size. We argue that making models entitycentric naturally fosters good architectural decisions. However, we also show that these models do not really build entity representations and that they make poor use of linguistic context. These negative results underscore the need for model analysis, to test whether the motivations for particular architectures are borne out in how models behave when deployed.1 Note the analogy with traditional models in formal linguistics like Discourse Representation Theory (Kamp and Reyle, 2013). 2 Source code for our model, the training procedure and the new dataset is published on https://github.com/ amore-upf/analysis-entity-centric-nns.3 https://competitions.codalab.org/ competitions/17310.

Distributional Models of Category Concepts Based on Names of Category Members

“…For this reason a distributional representation of an entire coreference chain (as in Adel & Schütze 2014; Clark & Manning 2016) could potentially supplant the use of names in an entity‐based approach to category representation. While coreference resolution has been the focus of much research interest, it is an open question whether current models are good enough to build entity representations (Aina, Silberer, Sorodoc, Westera, & Boleda, 2019; Clark & Manning, 2015).…”

Probing for Referential Information in Language Models

“…Several models (Henaff et al, 2019;Yang et al, 2017;Ji et al, 2017) were developed as an augmentation of RNN LMs to deal better with entities, with the implicit assumption that standard models do that poorly. Aina et al (2019) achieved good results on an entity-linking task, but showed that the network was not acquiring entity representations.…”

Exploring Neural Entity Representations for Semantic Information

“…Wiki2V, which is similarly contextbased, is also informed by its links to other entities which may provide additional information as we see in the next sections. The RNN's poor performance was also observed by Aina et al (2019), who saw low accuracy when probing an "entity-centric" RNN model for entity type information.…”

“…These techniques have similarly been applied to entity embeddings, though usually to limited extents. Entity type prediction has been among the most common task explored when proposing a new entity embedding method, in part because fine-grained entity type prediction is a common standalone task itself (Ling and Weld, 2012;Gupta et al, 2017;Yaghoobzadeh and Schütze, 2017;Aina et al, 2019;Chen et al, 2020). Recently, BERT-inspired techniques have been used to probe entity knowledge stored in pretrained language models through Cloze-style tasks, in which part of a fact about an entity is obscured and the model predicts the missing word(s) Petroni et al, 2019;Pörner et al, 2019;.…”