Modeling focus of attention for meeting indexing based on multiple cues

Abstract

A user's focus of attention plays an important role in human-computer interaction applications, such as a ubiquitous computing environment and intelligent space, where the user's goal and intent have to be continuously monitored. We are interested in modeling people's focus of attention in a meeting situation. We propose to model participants' focus of attention from multiple cues. We have developed a system to estimate participants' focus of attention from gaze directions and sound sources. We employ an omnidirectional camera to simultaneously track participants' faces around a meeting table and use neural networks to estimate their head poses. In addition, we use microphones to detect who is speaking. The system predicts participants' focus of attention from acoustic and visual information separately. The system then combines the output of the audio- and video-based focus of attention predictors. We have evaluated the system using the data from three recorded meetings. The acoustic information has provided 8% relative error reduction on average compared to only using one modality. The focus of attention model can be used as an index for a multimedia meeting record. It can also be used for analyzing a meeting.

Sampling techniques for audio-visual tracking and head pose estimation

“…In the first case, a generic tracker is used to locate the head, and then features extracted at this location are used to estimate the pose using any regression or classification tools. For example, [Stiefelhagen et al, 2002] used neural networks for such a task. Decoupling the tracking and pose estimation results in a computational cost reduction.…”

A Cognitive and Unsupervised Map Adaptation Approach to the Recognition of the Focus of Attention from Head Pose

“…In this meeting context, the goal of this paper is to analyze the correspondence between the head pose of people and their gaze in more natural and realistic scenarios than those previously considered [10,7], and propose methods to recognize the VFOA of people from their head pose. In contrast to previous approaches [10,7], our scenario involves people looking at slides or writing on a sheet of paper on the table.…”

“…In contrast to previous approaches [10,7], our scenario involves people looking at slides or writing on a sheet of paper on the table. As a consequence, people have more potential VFOA targets in our set-up (6 instead of 3 in the cited works), leading to more ambiguities between VFOA.…”

“…As a consequence, people have more potential VFOA targets in our set-up (6 instead of 3 in the cited works), leading to more ambiguities between VFOA. And, due to the physical placement of the VFOA targets, the identification of the VFOA can only be done using the complete head pose representation (pan and tilt), instead of just the head pan as in [10,7]. To recognize the VFOA of people, we investigate the use of an HMM model to segment pose observation sequences into VFOA temporal segments.…”

“…In addition, to account for the fact that people have their own head pose preferences for looking at the same given target, we adopted an unsupervised Maximum A Posteriori (MAP) scheme to adapt all the HMM parameters to individual people. This departs from [10], where the HMM parameters are directly learned on the test data after k-means initialization, an approach that could only work due to their simpler setting, as commented above.…”