Multi-modal sensing and analysis of poster conversations with smart posterboard

Abstract

Multi I . I N T R O D U C T I O NMulti-modal signal and information processing has been investigated primarily for intelligent human-machine interfaces, including smart phones, KIOSK terminals, and humanoid robots. Meanwhile, speech and imageprocessing technologies have been improved so much that their target now includes natural human-human behaviors, which are made without being aware of interface devices. In this scenario, sensing devices are installed in an ambient manner. Examples of this kind of direction include meeting capturing [1] and conversation analysis [2].We have been conducting a project which focuses on conversations in poster sessions, hereafter referred to as poster conversations [3,4]. Poster sessions have become a norm in many academic conventions and open laboratories because of the flexible and interactive characteristics. In most cases, however, paper posters are still used even in the ICT areas. In some cases, digital devices such as LCD and PC projectors are used, but they do not have sensing devices. Currently, many lectures in academic events are recorded and distributed via Internet, but recording of poster sessions is never done or even tried.Poster conversations have a mixture characteristics of lectures and meetings; typically a presenter explains his/her Academic Center for Computing and Media Studies, Kyoto University, Sakyo-ku, Kyoto 606-8501, JapanCorresponding author: T. Kawahara Email: kawahara@i.kyoto-u.ac.jp work to a small audience using a poster, and the audience gives feedbacks in real time by nodding and verbal backchannels, and occasionally makes questions and comments. Conversations are interactive and also multi-modal because participants are standing and moving unlike in meetings. Another good point of poster conversations is that we can easily make a setting for data collection which is controlled in terms of familiarity with topics and other participants and yet is "natural and real".The goal of this study is signal-level sensing and highlevel analysis of human interactions. Specific tasks include face detection, eye-gaze detection, speech separation, and speaker diarization. These will realize a new indexing scheme of poster session archives. For example, after a long session of poster presentation, we often want to get a short review of the question-answers and feedbacks from the audience. We also investigate high-level indexing of which segment was attractive and/or difficult for the audience to follow. This will be useful in speech archives because people would be interested in listening to the points other people liked. However, estimation of the interest and comprehension level is apparently difficult and largely subjective. Therefore, we turn to speech acts which are observable and presumably related with these mental states. One is prominent reactive tokens signaled by the audience and the other is questions raised by them. Prediction of these speech acts from multimodal behaviors is expected to approximate the estimation of the interest and comprehension...

An Introduction to Conversation Analysis

“…Even in quite sophisticated data collection approaches, capturing eye gaze accurately may be problematic. Kawahara, et al's (2016) study of audiences at poster presentations used a multiple forms of data collection involving a camera, a microphone array and motion sensing to detect audience members' faces and track their eye gaze. While this data collection method supplemented the video, they still needed to approximate eye gaze from head movement.…”

Situated language understanding for a spoken dialog system within vehicles

“…There are a number of studies that have analyzed the relationship between gaze and user intention, such as user focus (Yonetani et al, 2010), preference (Kayama et al, 2010), and reference expression understanding , and between gaze and turn-taking (Jokinen et al, 2010;Kawahara, 2012). Nakano et al (2013) used face direction for addressee identification.…”

Engagement recognition by a latent character model based on multimodal listener behaviors in spoken dialogue

“…On the other hand, the users’ internal states are difficult to define and measure objectively. Many researchers have proposed recognition models for various kinds of internal states such as the level of interest [7,8], understanding [9], and emotion [10–12]. From the perspective of the relationship between dialogue participants (i.e.…”