Learning to Recognize Transient Sound Events using Attentional Supervision

Abstract

Making sense of the surrounding context and ongoing events through not only the visual inputs but also acoustic cues is critical for various AI applications. This paper presents an attempt to learn a neural network model that recognizes more than 500 different sound events from the audio part of user generated videos (UGV). Aside from the large number of categories and the diverse recording conditions found in UGV, the task is challenging because a sound event may occur only for a short period of time in a video clip. Our model specifically tackles this issue by combining a main subnet that aggregates information from the entire clip to make clip-level predictions, and a supplementary subnet that examines each short segment of the clip for segment-level predictions. As the labeled data available for model training are typically on the clip level, the latter subnet learns to pay attention to segments selectively to facilitate attentional segment-level supervision. We call our model the M&mnet, for it leverages both “M”acro (clip-level) supervision and “m”icro (segment-level) supervision derived from the macro one. Our experiments show that M&mnet works remarkably well for recognizing sound events, establishing a new state-of-theart for DCASE17 and AudioSet data sets. Qualitative analysis suggests that our model exhibits strong gains for short events. In addition, we show that the micro subnet is computationally light and we can use multiple micro subnets to better exploit information in different temporal scales.

A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling

“…Our system closely matches the system of Yu et al [23], and outperforms all other systems by a large margin. [27,9] 1M 0.314 0.959 2.452 Kumar [28] 22k 0.213 0.927 Shah [18] 22k 0.229 0.927 Wu [29] 22k 0.927 Kong [22] 2M 0.327 0.965 2.558 Yu [23] 2M 0.360 0.970 2.660 Chen [30] 600k 0.316 Chou [31] 1M 0.327 0.951 [23] uses multi-level attention: attention layers are built upon multiple hidden layers, whose outputs are concatenated and further processed by a fully connected layer to yield a recording-level prediction. No frame-level predictions at all are made in this process.…”

Weakly Labelled AudioSet Tagging With Attention Neural Networks

“…Attention neural networks have been proposed for AudioSet tagging in [15,16]. Later, a cliplevel and segment-level model with attention supervision was proposed in [36].…”

Self-supervised Attention Model for Weakly Labeled Audio Event Classification

“…Strongly supervised attention loss: Existing attention models for weakly labeled AEC are usually trained by minimizing loss between clip-level labels, y ∈ {0, 1} C , and the clip-level predictions [12], [13]. The attention matrix learned in this process will focus on the most relevant and discriminative parts of the audio clip for prediction.…”

“…Recently, attention scheme has been applied to weakly labeled AEC. The attention mechanism helps a model focus on subsections of audio which contribute to the classification while ignoring the irrelevant instances such as background noises [11], [12].…”