Learning to Separate Object Sounds by Watching Unlabeled Video

Abstract

Perceiving a scene most fully requires all the senses. Yet modeling how objects look and sound is challenging: most natural scenes and events contain multiple objects, and the audio track mixes all the sound sources together. We propose to learn audio-visual object models from unlabeled video, then exploit the visual context to perform audio source separation in novel videos. Our approach relies on a deep multi-instance multi-label learning framework to disentangle the audio frequency bases that map to individual visual objects, even without observing/hearing those objects in isolation. We show how the recovered disentangled bases can be used to guide audio source separation to obtain better-separated, object-level sounds. Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video. We obtain state-of-the-art results on visuallyaided audio source separation and audio denoising. Our video results:

Recursive Visual Sound Separation Using Minus-Plus Net

“…The most related works are [28] and [10]. In [10], a convolutional network is used to predict the type of objects appeared in the video, and Non-negative Matrix Factorization [9] is used to extract a set of basic components. The association between each object and each basic component will be estimated via a Multi-Instance Multi-Label objective.…”

“…This would result in a feature tensor of size T × (H/16) × (W/16) × k. In both training and testing, this feature tensor will be reduced into a vector to represent the visual content by performing max pooling along the first three dimensions. On top of this solo video collection, we then follow the Mix-and-Separate strategy as in [28,10] to construct the mixed video/sound data, where each sample mixes n videos, called a mix-n sample.…”

“…Existing works [10,28] on visual sound separation mainly separate each sound independently. They assume either fixed types or fixed numbers of sounds, separating sounds independently.…”

To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations

“…A related topic is generating speech by measuring vibrations in a video [14]. Follow up works include separating input audio signals into a set of components that corresponds to different objects in the given video [20], and separating audio corresponding to each pixel [46].…”

Machine learning for music genre: multifaceted review and experimentation with audioset

“…Xu et al (2017) employ AudioSet for weakly supervised audio event detection, whereas Jansen et al (2017) extract semantic representations from non-speech audio following an unsupervised approach. Since the dataset is intended for general purpose audio event classification, it is suitable for a variety of problems related to audio, such as music o video processing (Gao et al, 2018;Zhou et al, 2018).…”