Sound Event Detection of Weakly Labelled Data With CNN-Transformer and Automatic Threshold Optimization

Abstract

Sound event detection (SED) is a task to detect sound events in an audio recording. One challenge of the SED task is that many datasets such as the Detection and Classification of Acoustic Scenes and Events (DCASE) datasets are weakly labelled. That is, there are only audio tags for each audio clip without the onset and offset times of sound events. We compare segment-wise and clip-wise training for SED that is lacking in previous works. We propose a convolutional neural network transformer (CNN-Transfomer) for audio tagging and SED, and show that CNN-Transformer performs similarly to a convolutional recurrent neural network (CRNN). Another challenge of SED is that thresholds are required for detecting sound events. Previous works set thresholds empirically, and are not an optimal approaches. To solve this problem, we propose an automatic threshold optimization method. The first stage is to optimize the system with respect to metrics that do not depend on thresholds, such as mean average precision (mAP). The second stage is to optimize the thresholds with respect to metrics that depends on those thresholds. Our proposed automatic threshold optimization system achieves a state-of-the-art audio tagging F1 of 0.646, outperforming that without threshold optimization of 0.629, and a sound event detection F1 of 0.584, outperforming that without threshold optimization of 0.564.

Impact of Sound Duration and Inactive Frames on Sound Event Detection Performance

“…We have compared the SED performance of our methods with those of other conventional methods, such as SED using a α min-max sub- sampling method within CRNN [18], the batch dice loss-based SED [23,27], multitask learning of SED and sound activity detection [28], and Transformer-based SED [12,13]. As the Transformerbased SED, we used three CNN layers with the same structure as the CNN-BiGRU, followed by two Transformer encoder layers and two dense layers.…”

“…For SED, many methods using neural networks, such as a convolutional neural network (CNN) [9], recurrent neural network (RNN) [10], convolutional recurrent neural network (CRNN) [11], and Transformer-based neural network [12,13], have been proposed. In these methods, an audio clip is segmented into short time frames (e.g., 40 ms frames), and each frame is regarded as one data sample for model training and evaluation.…”

Semi-Supervised Training of Transformer and Causal Dilated Convolution Network with Applications to Speech Topic Classification

“…However, these models discard the temporal order of frame-level features in their construction, leading to unsatisfactory final results. Different scholars have adopted different solutions to this type of time series information loss problem; for example, Pablo used End-to-End neural networks to solve this type of problem [20], Kong proposed an attention model and explained this model in a novel probabilistic perspective [21]. In addition, he proposed a convolutional neural network converter (CNN-Transformer) for audio tagging and SED and showed that the performance of the CNN-Transformer is similar to that of the CRNN [22].…”

Multi-Scale Gated Attention for Weakly Labelled Sound Event Detection

“…All the experiments are repeated 3 times and we report the best result of each model. In the post-processing phase, we use the same post-processing method as in paper [15] to get framelevel predictions and clip-level predictions.…”