The Coordinated Interplay of Scene, Utterance, and World Knowledge: Evidence From Eye Tracking

Abstract

Two studies investigated the interaction between utterance and scene processing by monitoring eye movements in agent-action-patient events, while participants listened to related utterances. The aim of Experiment 1 was to determine if and when depicted events are used for thematic role assignment and structural disambiguation of temporarily ambiguous English sentences. Shortly after the verb identified relevant depicted actions, eye movements in the event scenes revealed disambiguation. Experiment 2 investigated the relative importance of linguistic/world knowledge and scene information. When the verb identified either only the stereotypical agent of a (nondepicted) action, or the (nonstereotypical) agent of a depicted action as relevant, verb-based thematic knowledge and depicted action each rapidly influenced comprehension. In contrast, when the verb identified both of these agents as relevant, the gaze pattern suggested a preferred reliance of comprehension on depicted events over stereotypical thematic knowledge for thematic interpretation. We relate our findings to language comprehension and acquisition theories.

Language comprehension in rich non-linguistic contexts: combining eye-tracking and event-related brain potentials

“…In the above case, looks to an object have been interpreted as reflecting that the listener is thinking about that object and has thus established reference to it. The rapid inspection of named objects also highlights that language comprehension is closely temporally coordinated with visual attention (e.g., Altmann & Kamide, 1999;Knoeferle & Crocker, 2006Tanenhaus et al, 1995). This coordination is robust even when language refers to things that were (but no longer are) in front of our eyes (Altmann, 2004), when language is abstract (Duñabeitia, Avilés, Afonso, Scheepers, & Carreiras, 2008), and when objects are mentioned in rapid succession (Andersson, Ferreira, & Henderson, 2011).…”

“…This approach could constrain the interpretation of individual measures. It could also pave the way for extending current accounts of visually situated language comprehension (e.g., Knoeferle & Crocker, 2006, which have largely been shaped by eye-movement results, with a description of the functional brain correlates implicated in visually situated language processing.…”

“…interpretation preferences. Preferential inspection pattern emerged, for instance, in a study by Knoeferle and Crocker (2006). When the verb (e.g., 'spy-on') in a sentence about an action between two characters was compatible with either an action performed by a non-stereotypical agent (a wizard spying) or a stereotypical agent depicted as performing a mismatching action (a detective serving food), listeners' gaze pattern revealed interpretation preferences.…”

“…However, these accounts, just like the above language-centric accounts, do not aim to model the nature of the interplay between comprehension processes, (visual attention) and information from the non-linguistic visual context (see Altmann & Mirković, 2009;Crocker, Knoeferle, & Mayberry, 2010;Knoeferle & Crocker, 2006Mayberry, Crocker, & Knoeferle, 2009).…”

Situated sentence processing: The coordinated interplay account and a neurobehavioral model

“…We outline the Coordinated Interplay Account (CIA; Knoeferle & Crocker, 2006), and review both behavioral and neuroscientific findings in support of its three processing stages: (i) incremental sentence interpretation, (ii) language-mediated visual attention, and (iii) the on-line influence of non-linguistic visual context. We then describe a recently developed connectionist model which both embodies the central CIA proposals and has been successfully applied in modeling a range of findings from the visual world paradigm.…”