Interaction recognition in wide areas using audiovisual sensors

Abstract

We present an event recognition framework to detect interactions among objects, for example people, using a network of cameras and associated microphone pairs. The complementarity of the video and audio modalities is exploited to cover wide areas. In particular, object movements in portions of the scene that are not covered by the cameras' fields of view are estimated using the input from microphones. After estimating trajectories using audio-visual features, we recognize interactions based on a Coupled Hidden Markov Model Maximum a Posteriori (CHMM-MAP) approach. The states of the CHMM are initialized via Gaussian Mixture Model (GMM) clustering on a multi-dimensional feature space. Evaluation and comparison with three alternative methods demonstrate the effectiveness of the proposed CHMM-MAP trained on multiple features on both synthetic and real data.

Video-Based Human Behavior Understanding: A Survey

“…Moreover, in order to increase the performance achieved by pure vision systems in real scenarios, multi-modal sensing should be considered [183], [184]. Examples include the integration of 2D laser scanners with cameras [183] or cameras and microphones [185].…”