Sound Event Detection by Multitask Learning of Sound Events and Scenes with Soft Scene Labels

Abstract

Sound event detection (SED) and acoustic scene classification (ASC) are major tasks in environmental sound analysis. Considering that sound events and scenes are closely related to each other, some works have addressed joint analyses of sound events and acoustic scenes based on multitask learning (MTL), in which the knowledge of sound events and scenes can help in estimating them mutually. The conventional MTL-based methods utilize one-hot scene labels to train the relationship between sound events and scenes; thus, the conventional methods cannot model the extent to which sound events and scenes are related. However, in the real environment, common sound events may occur in some acoustic scenes; on the other hand, some sound events occur only in a limited acoustic scene. In this paper, we thus propose a new method for SED based on MTL of SED and ASC using the soft labels of acoustic scenes, which enable us to model the extent to which sound events and scenes are related. Experiments conducted using TUT Sound Events 2016/2017 and TUT Acoustic Scenes 2016 datasets show that the proposed method improves the SED performance by 3.80% in F-score compared with conventional MTL-based SED.

DCASENET: An Integrated Pretrained Deep Neural Network for Detecting and Classifying Acoustic Scenes and Events

“…Scarce research on the combination of related sound classification tasks has been conducted [10,11,13,14]. Imoto et al [13] assumed that ASC and SED are related and performed them simultaneously using a multitask learning framework.…”

“…For instance, perceiving car horns and traffic sounds can be helpful for knowing that he/she is standing in a street. Imoto et al [13,14] explored the relation between ASC and SED, proposing DNNs to perform the two tasks simultaneously through a multi-task learning framework [15]. However, the integration of DNNs for related tasks remains in a preliminary stage because only pairs of two related tasks are investigated and motivations on the relationship of these tasks has not been explored.…”

Cross-Task Transfer for Geotagged Audiovisual Aerial Scene Recognition

“…Cross-task Transfer. Transferring the learned knowledge from one task to another related task has been approved as an effective way for better data modeling and messages correlating [6,2,14]. Aytar et al [2] proposed a teacher-student framework that transfers the discriminative knowledge of visual recognition to the representation learning task of sound modality via minimizing the differences in the distribution of categories.…”

“…Aytar et al [2] proposed a teacher-student framework that transfers the discriminative knowledge of visual recognition to the representation learning task of sound modality via minimizing the differences in the distribution of categories. Imoto et al [14] proposed a method for sound event detection by transferring the knowledge of scenes with soft labels. Gan et al [8] transferred the visual object location knowledge for auditory localization learning.…”

“…This multi-task technique is very common within one single modality, such as solving depth estimation, surface normal estimation and semantic segmentation from one single image [7], or recognizing acoustic scenes and sound events from audio [14]. We apply this idea to multi-modality, and implement with Equation (4), encouraging the multimodal model to learn the underlying relationship between the sound events and the scenes for solving the two tasks simultaneously.…”