Gaze and turn-taking behavior in casual conversational interactions

Abstract

Eye gaze is an important means for controlling interaction and coordinating the participants' turns smoothly. We have studied how eye gaze correlates with spoken interaction and especially focused on the combined effect of the speech signal and gazing to predict turn taking possibilities. It is well known that mutual gaze is important in the coordination of turn taking in two-party dialogs, and in this article, we investigate whether this fact also holds for three-party conversations. In group interactions, it may be that different features are used for managing turn taking than in two-party dialogs. We collected casual conversational data and used an eye tracker to systematically observe a participant's gaze in the interactions. By studying the combined effect of speech and gaze on turn taking, we aimed to answer our main questions: How well can eye gaze help in predicting turn taking? What is the role of eye gaze when the speaker holds the turn? Is the role of eye gaze as important in three-party dialogs as in two-party dialogue? We used Support Vector Machines (SVMs) to classify turn taking events with respect to speech and gaze features, so as to estimate how well the features signal a change of the speaker or a continuation of the same speaker. The results confirm the earlier hypothesis that eye gaze significantly helps in predicting the partner's turn taking activity, and we also get supporting evidence for our hypothesis that the speaker is a prominent coordinator of the interaction space. Such a turn taking model could be used in interactive applications to improve the system's conversational performance.

Multimodal corpus of multiparty conversations in L1 and L2 languages and findings obtained from it

“…We collected multimodal data in three-party conversation to investigate whether the role of eye gaze is as important in three-party conversation as in two-party dialogue (Jokinen et al 2013). Furthermore, we collected 80 multimodal datasets (20 freeflowing in Japanese, 20 free-flowing in English, 20 goal-oriented in Japanese, and 20 goal-oriented in English) from multiparty conversations to investigate the difference in eye gaze between L1 and L2 conversations.…”

“…We performed the annotation according to the MUMIN annotation scheme (Allwood et al 2007) used in our previous research for modeling turn-taking behaviors in L1 conversations (Jokinen et al 2013). The annotation features and values adopted in our previous research are listed in Table 2.…”

“…Cohen's kappa coefficients were 0.55, 0.14, and 0.34 for Dialogue Act, Head Movement, and Hand Movement, respectively, in our preliminary test. We decided to start making annotations by limiting them, in the first stage of the research, to the features that were reported to be important in monitoring conversations, such as utterances, eye gazes, and turn-taking activities (Jokinen et al 2013). These features also maintained high agreement among the annotators.…”

“…Besides speech recognition, motion capture and gesture recognition technology can be used for automatic analysis of multimodal data, and the developments in eye-tracker technology allow us to study gaze behavior in an objective manner. Many quantitative studies on human-human interaction have also reported that eye gaze plays an important role in monitoring conversation content and contributes to the performance of collaborative tasks requiring the understanding of communication partners (Boyle et al 1994;Clark and Krych 2004;Jokinen et al 2013). These findings on human-human interactions were mainly obtained from conversations held in the mother tongue (L1).…”

Visual interaction and conversational activity

“…Certain culturally-dependent politeness codes thus determine what the appropriate manner is, to focus our visual attention to a stranger and to start interacting with them. Many studies suggest that gazing is a social phenomenon and an important element when people are engaged in the interaction [4,6,7]. For instance, comparing face-to-face interactions with a full-size and a computer-screen size video interactions, [4] noticed that in all three situations, all the participants fixate their gaze on the eyes of their conversational partner, but that in case of a computer-screen size interactions, the participants also look at their partner as a whole and fixate their gaze at other body parts unlike in face-toface or life-size video interactions where this was rare.…”