Polyphonic Sound Event and Sound Activity Detection: A Multi-Task Approach

Abstract

Polyphonic Sound Event Detection (SED) in real-world recordings is a challenging task because of the dynamic polyphony level, intensity, and duration of sound events. Current polyphonic SED systems fail to model the temporal structure of sound events explicitly and instead attempt to look at which sound events are present at each audio frame. Consequently, the event-wise detection performance is much lower than the segment-wise detection performance. In this work, we propose a joint model approach to improve the temporal localization of sound events using a multi-task learning setup. The first task predicts which sound events are present at each time frame; we call this branch 'Sound Event Detection (SED) model', while the second task predicts if a sound event is present or not at each frame; we call this branch 'Sound Activity Detection (SAD) model'. We verify the proposed joint model by comparing it with a separate implementation of both tasks aggregated together from individual task predictions. Our experiments on the URBAN-SED dataset show that the proposed joint model can alleviate False Positive (FP) and False Negative (FN) errors and improve both the segment-wise and the event-wise metrics.

Impact of Sound Duration and Inactive Frames on Sound Event Detection Performance

“…We have compared the SED performance of our methods with those of other conventional methods, such as SED using a α min-max sub- sampling method within CRNN [18], the batch dice loss-based SED [23,27], multitask learning of SED and sound activity detection [28], and Transformer-based SED [12,13]. As the Transformerbased SED, we used three CNN layers with the same structure as the CNN-BiGRU, followed by two Transformer encoder layers and two dense layers.…”

Joint Analysis of Sound Events and Acoustic Scenes Using Multitask Learning

“…To further demonstrate the utility of the proposed method, we applied our method to the conventional method for SED [27], which is based on the multitask learning of SED and SAD. The experimental results show that the event detection performance of SED + SAD [27] is better than that of CRNN (event). Moreover, the F-score of event detection in proposed + SAD (β= 0.01) was improved by 1.4 percentage points compared with that of SED + SAD [27].…”

Sound Event Detection Based on Curriculum Learning Considering Learning Difficulty of Events

“…As the baseline model of SED, we used the convolutional neural network and bidirectional gated recurrent unit (CNN-BiGRU) [8]. Moreover, to verify the usefulness of the proposed method, we used a model combining SED and sound activity detection (SAD) based on multitask learning (MTL), referred to as "MTL of SED & SAD" [25], and a model combining SED and ASC, referred to as "MTL of SED & ASC" [13]. The sound activity detection is the mechanism of recognizing any active events in a time frame.…”