Joint Analysis of Acoustic Events and Scenes Based on Multitask Learning

Abstract

Acoustic event detection and scene classification are major research tasks in environmental sound analysis, and many methods based on neural networks have been proposed. Conventional methods have addressed these tasks separately; however, acoustic events and scenes are closely related to each other. For example, in the acoustic scene "office", the acoustic events "mouse clicking" and "keyboard typing" are likely to occur. In this paper, we propose multitask learning for joint analysis of acoustic events and scenes, which shares the parts of the networks holding information on acoustic events and scenes in common. By integrating the two networks, we expect that information on acoustic scenes will improve the performance of acoustic event detection. Experimental results obtained using TUT Sound Events 2016/2017 and TUT Acoustic Scenes 2016 datasets indicate that the proposed method improves the performance of acoustic event detection by 10.66 percentage points in terms of the F-score, compared with a conventional method based on a convolutional recurrent neural network.

DCASENET: An Integrated Pretrained Deep Neural Network for Detecting and Classifying Acoustic Scenes and Events

“…Scarce research on the combination of related sound classification tasks has been conducted [10,11,13,14]. Imoto et al [13] assumed that ASC and SED are related and performed them simultaneously using a multitask learning framework.…”

“…For instance, perceiving car horns and traffic sounds can be helpful for knowing that he/she is standing in a street. Imoto et al [13,14] explored the relation between ASC and SED, proposing DNNs to perform the two tasks simultaneously through a multi-task learning framework [15]. However, the integration of DNNs for related tasks remains in a preliminary stage because only pairs of two related tasks are investigated and motivations on the relationship of these tasks has not been explored.…”

Sound Event Detection by Multitask Learning of Sound Events and Scenes with Soft Scene Labels

“…However, some sound events and scenes are closely related, and the knowledge of sound events and scenes can help in estimating them mutually. Considering this idea, joint analysis of sound events and acoustic scenes based on MTL of SED and ASC has been proposed [18,19]. As shown in Fig.…”

“…where α indicates the weight of the scene loss. In particular, the previous work [19] showed that an MTL-based method achieves a better performance in detecting sound events than CRNN-based SED [12].…”

Sound Event Detection Based on Curriculum Learning Considering Learning Difficulty of Events

“…As an example of the relationship, "mouse clicking" occurs indoors such as "office," whereas, "car" tends to occurs outdoor such as "city center." On the basis of this idea, SED using the information on the acoustic scene [9][10][11] and the model combining SED and acoustic scene classification (ASC) [12][13][14][15][16] have been proposed. Heittola et al [10] have proposed the SED model using the results of the ASC, where the ASC model is trained in the first stage and then the SED model is trained in the second stage with the ASC results.…”