Integrating mechanisms of visual guidance in naturalistic language production

Abstract

Situated language production requires the integration of visual attention and linguistic processing. Previous work has not conclusively disentangled the role of perceptual scene information and structural sentence information in guiding visual attention. In this paper, we present an eye-tracking study that demonstrates that three types of guidance, perceptual, conceptual, and structural, interact to control visual attention. In a cued language production experiment, we manipulate perceptual (scene clutter) and conceptual guidance (cue animacy), and measure structural guidance (syntactic complexity of the utterance). Analysis of the time course of language production, before and during speech, reveals that all three forms of guidance affect the complexity of visual responses, quantified in terms of the entropy of attentional landscapes and the turbulence of scan patterns, especially during speech. We find that perceptual and conceptual guidance mediate the distribution of attention in the scene, whereas structural guidance closely relates to scan-pattern complexity. Furthermore, the eye-voice span of the cued object and its perceptual competitor are similar; its latency mediated by both perceptual and structural guidance. These results rule out a strict interpretation of structural guidance as the single dominant form of visual guidance in situated language production. Rather, the phase of the task and the associated demands of cross-modal cognitive processing determine the mechanisms that guide attention.

Anticipation in Real‐World Scenes: The Role of Visual Context and Visual Memory

“…The visual cognition literature shows that clutter influences search behavior, with clutter measures such as feature congestion (Rosenholtz, Li, & Nakano, 2007) correlating with search accuracy and fixation behavior (Henderson, Chanceaux, & Smith, 2009). It is therefore reasonable to assume that clutter has an effect on language-mediated eye-movements as they occur in the visual world paradigm 3 , as recently shown by Coco and Keller (2015a) The second, and perhaps most important, difference between clip-art images and real-world scenes is that the latter depict semantically coherent collections of objects, providing the viewer with contextual information which he/she can use to pro-actively allocate visual attention (see Figure 2 for an example). Contextual information restricts where objects typically occur (a table cannot float in the sky, a sofa is likely to be found indoors), as well as how objects relate to each other (monitors and keyboards tend to occur together, and in certain spatial configurations).…”

“…Eye-movements and speech are tightly time-locked (e.g., Coco & Keller, 2012, 2015aGriffin & Bock, 2000), indicating that the cognitive system is able to efficiently link visual information to linguistic input. This is a straightforward task in impoverished visual contexts such as the clip-art scene in Fig.…”

“…The visual cognition literature shows that clutter influences search behavior, with clutter measures such as feature congestion (Rosenholtz, Li, & Nakano, 2007) correlating with search accuracy and fixation behavior (Henderson, Chanceaux, & Smith, 2009). It is therefore reasonable to assume that clutter has an effect on language-mediated eye-movements as they occur in the visual world paradigm, 3 as recently shown by Coco and Keller (2015a).…”

“…Eye-movements and speech are tightly time-locked (e.g., Griffin & Bock, 2000;Coco & Keller, 2012, 2015a, indicating that the cognitive system is able to efficiently link visual information to linguistic input. This is a straightforward task in impoverished visual contexts such as the clip-art scene in Figure 1(a), where there are only four objects, making it easy to identify and fixate the edible object CAKE once the verb eat has been processed.…”

Information and Selforganization: A Unifying Approach and Applications

“…Here, for instance, saliency maps (attentional landscapes (e.g., [72,73]) can be established and their (information) entropy measured. This might open a way to connect our type of modelling with experimental findings by Coco and Keller [74] to mention but one recent example.…”