Easy Things First: Installments Improve Referring Expression Generation for Objects in Photographs

Abstract

Research on generating referring expressions has so far mostly focussed on "oneshot reference", where the aim is to generate a single, discriminating expression. In interactive settings, however, it is not uncommon for reference to be established in "installments", where referring information is offered piecewise until success has been confirmed. We show that this strategy can also be advantageous in technical systems that only have uncertain access to object attributes and categories. We train a recently introduced model of grounded word meaning on a data set of REs for objects in images and learn to predict semantically appropriate expressions. In a human evaluation, we observe that users are sensitive to inadequate object names -which unfortunately are not unlikely to be generated from low-level visual input. We propose a solution inspired from human task-oriented interaction and implement strategies for avoiding and repairing semantically inaccurate words. We enhance a word-based REG with contextaware, referential installments and find that they substantially improve the referential success of the system.

Obtaining referential word meanings from visual and distributional information: Experiments on object naming

“…Additional evaluation metrics, such as success rates in a human evaluation (cf. Zarrieß and Schlangen (2016)), would be an interesting direction for more detailed investigation here.…”

“…These features are then associated in a learning process with certain words, resulting in an association of colour features with colour words, spatial features with prepositions, etc., and based on this, these words can be interpreted with reference to the scene currently presented to the video feed. Whereas Roy's work still looked at relatively simple scenes with graphical objects, research on REG has recently started to investigate set-ups based on real-world images (Kazemzadeh et al, 2014;Gkatzia et al, 2015;Zarrieß and Schlangen, 2016;Mao et al, 2015). Importantly, the lowlevel visual features that can be extracted from these scenes correspond less directly to particular word classes.…”

“…Moreover, the visual scenes contain many different types of objects, which poses new challenges for REG. For instance, Zarrieß and Schlangen (2016) find that semantic errors related to mismatches between nouns (e.g. the system generates tree vs. man) are particularly disturbing for users.…”

“…the system generates tree vs. man) are particularly disturbing for users. Whereas Zarrieß and Schlangen (2016) propose a strategy to avoid object names when the systems confidence is low, we focus on improving the generation of object names, using distributional knowledge as an additional source. Similarly, Ordonez et al (2016) have studied the problem of deriving appropriate object names, or so-called entry-level categories, from the output of an object recognizer.…”

Is this a Child, a Girl or a Car? Exploring the Contribution of Distributional Similarity to Learning Referential Word Meanings

“…To factor out effects of compositionality and context that arise in reference generation or resolution, we measure how well a predictor for a word w is able to retrieve from a sampled test set objects that have been referred to by w (Schlangen et al, 2016;Zarrieß and Schlangen, 2016a) evaluate on full referring expressions).…”