Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications

Abstract

Visual events are usually accompanied by sounds in our daily lives. However, can the machines learn to correlate the visual scene and sound, as well as localize the sound source only by observing them like humans? To investigate its empirical learnability, in this work we first present a novel unsupervised algorithm to address the problem of localizing sound sources in visual scenes. In order to achieve this goal, a two-stream network structure which handles each modality with attention mechanism is developed for sound source localization. The network naturally reveals the localized response in the scene without human annotation. In addition, a new sound source dataset is developed for performance evaluation. Nevertheless, our empirical evaluation shows that the unsupervised method generates false conclusions in some cases. Thereby, we show that this false conclusion cannot be fixed without human prior knowledge due to the well-known correlation and causality mismatch misconception. To fix this issue, we extend our network to the supervised and semi-supervised network settings via a simple modification due to the general architecture of our two-stream network. We show that the false conclusions can be effectively corrected even with a small amount of supervision, i.e., semi-supervised setup. Furthermore, we present the versatility of the learned audio and visual embeddings on the cross-modal content alignment and we extend this proposed algorithm to a new application, sound saliency based automatic camera view panning in 360 • videos.

Conditioned Source Separation for Musical Instrument Performances

“…With the outbreak of deep learning techniques, the field of audio-visual learning has received a significant boost, especially the problems formulated in unsupervised and selfsupervised manners. Along this line of research there have been some works focused on representation learning with further applications in audio classification, action recognition and source localisation [18], [37], [38], [39], [40], [41], [42], [43]. Most of them combined features from two-stream networks (one sub-network for the audio and another one for the visual modality) either by concatenating them or by having an additional attention module.…”

“…Most of them combined features from two-stream networks (one sub-network for the audio and another one for the visual modality) either by concatenating them or by having an additional attention module. Some of them employed time synchrony for the samples of the same video [18], [44], while others learnt to extract features by identifying if the audio sample corresponded to a given visual data [18], [37], [40]. More recent work also focused on the usage of audio for distilling redundant visual information to reduce computational costs [41].…”

“…Different objective functions such as cross-entropy [38], [39], Kullback-Leibler divergence [37], [41], contrastive [18] or triplet [40] losses were exploited in audio-visual deep learning. Distinctively, Korbar et al [18] used curriculum learning by first training the network with easy examples (correspondence was defined as being sampled from the same video) and then with hard/superhard examples (correspondence was defined as time-synchrony with/without time shift within the same video).…”

Low-Cost Network Scheduling of 3D-CNN Processing for Embedded Action Recognition

“…It is also possible to reduce the complexity of the action recognition process by exploiting the additional side information. For example, the sound information or pre-calculated features, which are normally existed in the compressed video data, can be utilized to provide the additional information for enabling the cliplevel processing [37] or selecting the dominant clips strongly related to the actions [38], [39], increasing the recognition accuracy while even reducing the computational complexity.…”

HCMS: Hierarchical and Conditional Modality Selection for Efficient Video Recognition

“…Audio cue is also utilized to improve video recognition performance [15,45,47]. These multimodal inputs are used in other video tasks such as self-supervised learning in videos [1,25], sound localization [32,36] and generation [54,55] from videos as well. Existing fusion approaches typically use fixed fusion weights for all samples.…”