Visual interaction and conversational activity

Abstract

In addition to the contents of their speech, people who are engaged in a conversation express themselves in many nonverbal ways. This means that people interact and are attended to even when they are not speaking. In this pilot study, we created an experimental setup for a three-party interactive situation where one of the participants remained silent throughout the session, and the gaze of one of the active subjects was tracked. The eyetracked subject was unaware of the setup. The pilot study used only two test subjects, but the results provide some clues towards estimating how the behavior and activity of the non-speaking participant might affect other participants' conversational activity and the situation itself. We also found that the speaker's gaze activity is different in the beginning of the utterance than at the end of the utterance, indicating that the speaker's focus of attention towards the partner differs depending on the turn taking situation. Using the experience gained in this trial, we point out several things to consider that might help to avoid pitfalls when designing a more extensive study into the subject.

Rushing to Judgement: How do Laypeople Rate Caller Engagement in Thin-Slice Videos of Human–Machine Dialog?

“…For example, Zhou et al presented a non-task oriented engagementaware dialog system which was trained by having 2 expert annotators rate how engaging different strategies were [10]. Multiple research studies have examined the annotation and prediction of user engagement in videos of multi-party dialog, and have typically relied on gold-standard annotations rated by a few annotators (see for instance [11,12,13]). Such analysis and prediction of engagement and other learner states are also critical to the design and development of intelligent tutors and computerassisted language learning (CALL) systems in the education domain [14,15].…”

A gaze-based method for relating group involvement to individual engagement in multimodal multiparty dialogue

“…Levitski et al [8] carried out an analysis on a three-party conversation corpus. In their set-up they tracked the eyegaze of one participant.…”

“…First of all, our analysis is based on a eight party conversation rather than a 4 or 5 party conver- [13,5,1] and [8]. Third, differently from [3] and [8], our individual engagement annotations are based on the rankings of the participants themselves and we are using a predefined annotation scheme rather than relying solely on the thirdparty annotators' intuitions. However, similarly to [3], we are not using fixed window length for the group involvement annotations.…”

Multimodal Open-Domain Conversations with the Nao Robot

“…Gaze-tracking is important in order to manage smooth turn-taking [7,12] and to get feedback about the partner's interest in the topic. As humans direct their gaze towards objects of interest, it is useful if the robot can infer where the partner's attention is focussed, and if they are still interested in what it is presenting.…”