A Method Based on Dual Cross-Modal Attention and Parameter Sharing for Polyphonic Sound Event Localization and Detection

Abstract

Sound event localization and detection (SELD) is a joint task that unifies sound event detection (SED) and direction-of-arrival estimation (DOAE). The task has become such a popular topic that it was introduced into the challenge on Detection and Classification of Acoustic Scenes and Events (DCASE) Task3 in 2019. In this paper, we propose a method based on dual cross-modal attention (DCMA) and parameter sharing to simultaneously detect and localize sound events. In particular, the DCMA-based decoder commonly used for multiple predictions efficiently learns the associations between SED and DOAE features by exchanging SED and DOAE information in the process of attention, in addition to the encoder with parameter sharing. Furthermore, acoustic features that have not been usually used in the SELD task are additionally adopted to improve the performance, and data augmentation techniques of the mixup to simulate polyphonic events and channel rotation for spatial augmentation are conducted for this task. Experimental results demonstrate that our efficient model using one common decoder block based on the DCMA to predict multiple events in the track-wise output format is effective for the SELD task with up to three overlapping events.

Sound events detection and localization improvements by using Gammatone filters and temporal convolutional neural networks

“…Transformers are able to capture time-dependent features in data, without the use of recurrent networks as GRU or LSTM. This approach was adapted to the SELD task in (Lee et al, 2021;Park et al, 2021a).…”