Audiovisual event detection towards scene understanding

Abstract

Acoustic events produced in meeting environments may contain useful information for perceptually aware interfaces and multimodal behavior analysis. In this paper, a system to detect and recognize these events from a multimodal perspective is presented combining information from multiple cameras and microphones. First, spectral and temporal features are extracted from a single audio channel and spatial localization is achieved by exploiting crosscorrelation among microphone arrays. Second, several video cues obtained from multi-person tracking, motion analysis, face recognition, and object detection provide the visual counterpart of the acoustic events to be detected. A multimodal data fusion at score level is carried out using two approaches: weighted mean average and fuzzy integral. Finally, a multimodal database containing a rich variety of acoustic events has been recorded including manual annotations of the data. A set of metrics allow assessing the performance of the presented algorithms. This dataset is made publicly available for research purposes.

Improving detection of acoustic events using audiovisual data and feature level fusion

“…The comparison of two baseline detection systems for dataset of isolated AEs is presented in table 1. The first system (Bas1) exploits the classical approach, where all AEs are detected inside a unique AED system [7]. The second (Bas2) exploits the one-against-all topology described in the previous section.…”

“…This idea was first presented in [6] where the detection of footsteps was improved by exploiting the velocity information obtained from a video-based person-tracking system. Further improvement has been achieved by the authors in [7] where the concept of multimodal AED is extended to detect and recognize the set of 11 AEs. In that work, not only video information but also acoustic source localization information was considered.…”

“…In this paper, we compare that previous approach [7] with a feature-level fusion strategy and present results for AED on new and rather spontaneous data, where the temporal overlaps of sounds take place more frequently. Additionally, a new HMM-GMM based AED system structure is used which considers each class separately and uses a one-against-all strategy for training.…”