Towards Generating Colour Terms for Referents in Photographs: Prefer the Expected or the Unexpected?

Abstract

Colour terms have been a prime phenomenon for studying language grounding, though previous work focussed mostly on descriptions of simple objects or colour swatches. This paper investigates whether colour terms can be learned from more realistic and potentially noisy visual inputs, using a corpus of referring expressions to objects represented as regions in real-world images. We obtain promising results from combining a classifier that grounds colour terms in visual input with a recalibration model that adjusts probability distributions over colour terms according to contextual and object-specific preferences.

Is this a Child, a Girl or a Car? Exploring the Contribution of Distributional Similarity to Learning Referential Word Meanings

“…Similarly, distributional similarities between colors seem to be misleading rather than helpful, cf. (Zarrieß and Schlangen, 2016b) for a study on color adjectives on the same corpus. This effect seems to be related to findings on antonyms in distributional modeling (Nguyen et al, 2016).…”

Do Hesitations Facilitate Processing of Partially Defective System Utterances? An Exploratory Eye Tracking Study

“…The dialogue system's task is to generate expressions referring to objects in real-world images, intending to identify these objects to a human listener. The system's underlying generation component predicts words directly from low-level visual input representations of the target object defined via a bounding box in the image, based on the approach in [6,7]. As illustrated in Figure 1, this can lead to partially defective utterances being generated, due to imperfect visual language grounding [7].…”

“…Thus, it has been argued, that dialogue systems interacting with users in real-world environments need principled communicative mechanisms for dealing with uncertainties, perceptual mismatches, and potential misunderstanding [14,15,16,17]. In this study, we look at (potentially less disturbing) defective color terms, which turn out to be hard to predict for objects in real-world images as well [6]. Thus, for compiling the materials of our experiment, we used color terms predicted for objects in images by [6]'s model which we identified as defective based on annotated color terms in the training set of the model.…”

“…In this study, we look at (potentially less disturbing) defective color terms, which turn out to be hard to predict for objects in real-world images as well [6]. Thus, for compiling the materials of our experiment, we used color terms predicted for objects in images by [6]'s model which we identified as defective based on annotated color terms in the training set of the model. This allows us to investigate whether conversational synthesis can lead to processing advantages for the listener for realistic, but controlled errors in system output.…”

Referring Expression Generation under Uncertainty: Algorithm and Evaluation Framework

“…While there have been previous approaches to generating referring expressions (REs) under uncertainty, those algorithms have been explicitly designed to refer to objects in visual scenes, and as such are tightly integrated with visual classifiers (Zarrieß and Schlangen, 2016;Roy, 2002;Meo et al, 2014). This is problematic for least two reasons: First, intelligent agents may need to generate REs for a much wider class of entities than those appearing in a visual scene (e.g., agents, locations, ideas, utterances), which may not be possible if an REG algorithm is tightly coupled with visual classifiers.…”