Referring Expression Generation under Uncertainty: Algorithm and Evaluation Framework

Abstract

For situated agents to effectively engage in natural-language interactions with humans, they must be able to refer to entities such as people, locations, and objects. While classic referring expression generation (REG) algorithms like the Incremental Algorithm (IA) assume perfect, complete, and accessible knowledge of all referents, this is not always possible. In this work, we show how a previously presented consultant framework (which facilitates reference resolution when knowledge is uncertain, heterogeneous and distributed) can be used to extend the IA to produce DIST-PIA, a domain-independent algorithm for REG under uncertain, heterogeneous, and distributed knowledge. We also present a novel framework that can be used to evaluate such REG algorithms without conflating the performance of the algorithm with the performance of classifiers it employs.

Specificity measures and reference

“…While gradable properties have been addressed in the REG literature (van Deemter, 2006), the implications of graduality for referential success remain under-explored and the complex interactions between the evaluation of the properties of an en-tity and the visual context it is in have only recently begun to benefit from principled accounts (Yu et al, 2016;Williams and Scheutz, 2017). The account of referential success given here is based on the concept of specificity, a measure of the identifiability of an entity based on the possibility distribution associated with its properties.…”

“…This has become increasingly evident in work that has sought solutions to the REG problem in naturalistic scenes, as part of a broader research focus on the visionlanguage interface (Kazemzadeh et al, 2014;Mao et al, 2016;Yu et al, 2016). However, context dependence is also a central concern for approaches to REG that assume a more structured input representation where entities and their properties are available, but the extent to which a property applies to a referent is not necessarily an allor-none decision (Horacek, 2005;van Deemter, 2006;Turner et al, 2008;Williams and Scheutz, 2017). Under these conditions, it is no longer possible to assume that properties are crisp or Boolean, or even that both sender and receiver necessarily assume the same semantics for those properties.…”

Generating Spatial Referring Expressions in a Social Robot: Dynamic vs. Non-ambiguous

“…This also makes for a description that is more useful to the listener. Evaluation frameworks for generation algorithms often have a bias toward a complete and non-ambiguous description by being based upon a single direction of communication (Williams and Scheutz, 2017). This highlights the fact that we often take the description as the goal, and not the communication that a description is for Krahmer and Van Deemter (2012).…”

Towards Generating Spatial Referring Expressions in a Social Robot: Dynamic vs Non-Ambiguous

“…Evaluation frameworks for generation algorithms are often based upon a single direction of communication [5]. However communicating the location of an object to someone else is often a two way communication [6].…”