Acoustic Event Detection Based on Feature-Level Fusion of Audio and Video Modalities

Abstract

Acoustic event detection (AED) aims at determining the identity of sounds and their temporal position in audio signals. When applied to spontaneously generated acoustic events, AED based only on audio information shows a large amount of errors, which are mostly due to temporal overlaps. Actually, temporal overlaps accounted for more than 70% of errors in the realworld interactive seminar recordings used in CLEAR 2007 evaluations. In this paper, we improve the recognition rate of acoustic events using information from both audio and video modalities. First, the acoustic data are processed to obtain both a set of spectrotemporal features and the 3D localization coordinates of the sound source. Second, a number of features are extracted from video recordings by means of object detection, motion analysis, and multicamera person tracking to represent the visual counterpart of several acoustic events. A feature-level fusion strategy is used, and a parallel structure of binary HMM-based detectors is employed in our work. The experimental results show that information from both the microphone array and video cameras is useful to improve the detection rate of isolated as well as spontaneously generated acoustic events.

Learning Representations for Nonspeech Audio Events Through Their Similarities to Speech Patterns

“…Finally, our work bears some resemblances to those exploring additional data sources (e.g. multiple channels [44], multiple modalities [45]) to augment the nonspeech audio event analysis. However, their main goal is to compensate for low signal-to-noise-ratio and overlapping signals.…”

“…We used audio events of three independent datasets, including UPC-TALP [45], Freiburg-106 [16], and NAR [17], as the target nonspeech signals. These datasets are recorded in different environments, and hence, differ in reverberation characteristics.…”

“…• The UPC-TALP dataset [45] was recorded in a meetingroom environment. This dataset is multi-channel and multimodal (i.e.…”

Sound event recognition in unstructured environments using spectrogram image processing

“…Another area that has attracted research is focussed on the detection and classification of sounds in a meeting room environment [35][36][37][38][39]. It is also one of the only areas of SER research that has a standardised database for comparing the performance of competing systems.…”

Supervised model training for overlapping sound events based on unsupervised source separation

“…In the event detection stage, overlapping events can be detected with multiple iterative detection passes and by excluding already detected events from the following detection iterations until the desired amount of overlapping events have been reached [15]. In addition to these approaches, multiple audio signals and sound source localization methods along with video based methods can be used to better handle overlapping event in the detection [16].…”