Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension

Abstract

We investigated the coupling between a speaker's and a listener's eye movements. Some participants talked extemporaneously about a television show whose cast members they were viewing on a screen in front of them. Later, other participants listened to these monologues while viewing the same screen. Eye movements were recorded for all speakers and listeners. According to cross-recurrence analysis, a listener's eye movements most closely matched a speaker's eye movements at a delay of 2 sec. Indeed, the more closely a listener's eye movements were coupled with a speaker's, the better the listener did on a comprehension test. In a second experiment, low-level visual cues were used to manipulate the listeners' eye movements, and these, in turn, influenced their latencies to comprehension questions. Just as eye movements reflect the mental state of an individual, the coupling between a speaker's and a listener's eye movements reflects the success of their communication.

Pictures and Spoken Descriptions Elicit Similar Eye Movements During Mental Imagery, Both in Light and in Complete Darkness

“…In a study of simultaneous descriptions of the same stimulus picture, Holsanova (2001, p. 104f) found that eye-voice latencies (i.e., the time from when an object is mentioned until the eye moves to the corresponding location) typically range between 2 and 4 sec. Similar results were found by Richardson and Dale (2005), who showed that participants who listen to a speaker talking about present picture elements are most likely to look at the Fig. 1.…”

Gaze allocation in a dynamic situation: Effects of social status and speaking

“…When participants listen to spoken sentences they recognize objects named in the sentence and move their eyes to these objects a few hundred milliseconds after the beginning of the word (Allopenna, Magnuson, & Tanenhaus, 1998). In their study of gaze coordination, Richardson and Dale (2005) found that when viewing a picture of characters from a TV show, a listener's eye movements were best aligned with those of the speaker after a 2 s delay. In these studies, presumably, participants take time to process what they hear and plan appropriate eye movements.…”

“…This technique analyses the correlation between two signals over time, and it provides a correlation coefficient when the two signals are perfectly aligned (the ''zero lag"), as well as when one signal is shifted relative to the other (see Richardson & Dale, 2005, for a similar approach). In our case, we computed a cross correlation for each target, in each clip, between the record of speaking and the proportion of observers watching that clip who were fixating that target.…”

Movement constraints on interpersonal coordination and communication.

“…Together, these studies show that gaze-based measures of joint attention index effective communication and shared cognitive states. Given previous findings linking movement with cognition (Freeman et al, 2011;McKinstry et al, 2008;Spivey et al, 2005), the degree to which joint attention is captured by gaze coordination (Richardson & Dale, 2005) suggests a functional link between movement coordination and cognitive coordination that may be amenable to experimental manipulation. However, to our knowledge, there has been no study that has combined measurements of gaze coordination and interpersonal postural coordination, neither is there an account of the effects of changes in movement coordination on either shared attentional states or on communication.…”