Eye gaze and viewpoint in multimodal interaction management

Abstract

In this paper, we present an embodiment perspective on viewpoint by exploring the role of eye gaze in face-to-face conversation, in relation to and interaction with other expressive modalities. More specifically, we look into gaze patterns, as well as gaze synchronization with speech, as instruments in the negotiation of participant roles in interaction. In order to obtain fine-grained information on the different modalities under scrutiny, we used the InSight Interaction Corpus (Brône, Geert & Bert Oben. 2015. Insight Interaction: A multimodal and multifocal dialogue corpus. Language Resources and Evaluation 49, 195–214.). This multimodal video corpus consists of two- and three-party interactions (in Dutch), with head-mounted scene cameras and eye-trackers tracking all participants’ visual behavior, providing a unique ‘speaker-internal’ perspective on the conversation. The analysis of interactional sequences from the corpus (dyads and triads) reveals specific patterns of gaze distribution related to the temporal organization of viewpoint in dialogue. Different dialogue acts typically display specific gaze events at crucial points in time, as, e.g., in the case of brief gaze aversion associated with turn-holding, and shared gaze between interlocutors at the critical point of turn-taking. In addition, the data show a strong correlation and temporal synchronization between eye gaze and speech in the realization of specific dialogue acts, as shown by means of a series of cross-recurrence analyses for specific turn-holding mechanisms (e.g., verbal fillers co-occurring with brief moments of gaze aversion).

The Role of Eye Gaze in Regulating Turn Taking in Conversations: A Systematized Review of Methods and Findings

“…There was great variation between studies in the level of detail provided with multiple papers omitting important details about design and participants or the rationale for their methodological decisions. The studies varied in size from five to 69 participants ( Table 3 ), but only 11 studies reported participant's ages (Lamb, 1981 ; Harrigan and Steffen, 1983 ; Harrigan, 1985 ; Egbert, 1996 ; Rossano et al, 2009 ; Eberhard and Nicholson, 2010 ; Cummins, 2012 ; Jokinen et al, 2013 ; Ho et al, 2015 ; Holler and Kendrick, 2015 ; Brône et al, 2017 ), which ranged between 18 and 65 years. Fourteen of the 29 studies examined conversation in dyads, 11 looked at triads, six studied multiparty conversations, and one did not report the number of interactants ( Table 3 ).…”

“…All 29 studies used video recording to capture eye gaze during conversation, however nine did not specify how many cameras were used (Beattie, 1978 , 1979 ; Rutter et al, 1978 ; Goodwin, 1980 ; Harrigan, 1985 ; Egbert, 1996 ; Lerner, 2003 ; Park, 2015 ; Blythe et al, 2018 ). Seven studies used one camera for each participant (Lamb, 1981 ; Bavelas et al, 2002 ; Eberhard and Nicholson, 2010 ; Cummins, 2012 ; Ho et al, 2015 ; Holler and Kendrick, 2015 ; Ijuin et al, 2018 ), three studies used one camera for the whole group interaction (Kendon, 1967 ; Harrigan and Steffen, 1983 ; Streeck, 2014 ), seven studies video recorded both each participant plus the whole group interaction (Kalma, 1992 ; Novick et al, 1996 ; Brône et al, 2017 ; Kendrick and Holler, 2017 ; Auer, 2018 ; Weiss, 2018 ; Zima et al, 2019 ), two studies only video recorded two out of three participants and eye tracked the third participant (Jokinen et al, 2009 , 2013 ), one study used two cameras to capture interactions in Italian language and only one camera to capture interactions in Tzeltal and Yeli Dnye languages Rossano et al ( 2009 ). Eleven studies used camera-based eye tracking technology ( Table 4 ), which permits investigators to measure participant's visual behavior by detecting and tracking movement of different parts of the eye (see review: Morimoto and Mimica, 2005 ).…”

Optimizing turn-taking in interpreter-mediated therapy: on the importance of the interpreter’s speaking space

“…Recent years have witnessed a growing interest in nonverbal resources for self-selection in conversational interaction. Studies have shown that incipient speakers tend to gaze away just before starting to speak, which appears to signal to the interlocutors that they are about to take the turn (Kendon, 1967;Duncan, 1972;Brône et al, 2017). Even by using gestures speakers can make a claim for speakership publicly visible.…”

Gaze and overlap resolution in triadic interactions

“…Towards the end of TCUs, gaze at a given co-participant is a means to select her as next-speaker while speakers' gaze aversion signals a wish to continue (cf. also Lerner, 2003;Stivers and Rossano, 2010;Streeck, 2014;Weiß and Auer, 2016;Auer, 2018;Brône et al, 2017). (ii) Gaze aversion at TRPs by gaze-selected next speakers, in turn, may either be an interactional cue that they are about to begin to speak (Kendon, 1967;Ho et al, 2015) or, quite the opposite, a signal of their wish not to take the floor.…”