Co-Separating Sounds of Visual Objects

Abstract

Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel. Current methods for visually-guided audio source separation sidestep the issue by training with artificially mixed video clips, but this puts unwieldy restrictions on training data collection and may even prevent learning the properties of "true" mixed sounds. We introduce a co-separation training paradigm that permits learning object-level sounds from unlabeled multi-source videos. Our novel training objective requires that the deep neural network's separated audio for similar-looking objects be consistently identifiable, while simultaneously reproducing accurate videolevel audio tracks for each source training pair. Our approach disentangles sounds in realistic test videos, even in cases where an object was not observed individually during training. We obtain state-of-the-art results on visuallyguided audio source separation and audio denoising for the MUSIC, AudioSet, and AV-Bench datasets.where M V 1 and M V 2 are the ground-truth spectrogram ratio masks for the two videos, respectively. Namely,

Music Gesture for Visual Sound Separation

“…Ephrat et al [12] and Owens et al [38] proposed to used vision to improve the quality of speech separation. Xu et al [53] and Gao et al [19] further improved the models with recursive models and co-separation loss. Those works all demonstrated how semantic appearances could help with sound separation.…”

“…• NMF [52] is a well established pipeline for audio-only source separation based on matrix factorization; • Deep Separation [9] is a CNN-based audio-only source separation approach; • MIML [17] is a model that combines NMF decomposition and multi-instance multi-label learning; • Sound of Pixels [57] is a pioneering work that uses vision for sound source separations; • Co-separation [19] devices a new model that incorporates an object-level co-separation loss into the mixand-separate framework [57]; • Sound of Motions [56] is a recently proposed selfsupervised model which leverages trajectory motion cues. We adopt the blind separation metrics, including signalto-distortion ratio (SDR), and signal-to-interference ratio (SIR) to quantitatively compare the quality of the sound separation.…”

“…Our model is built on the mix-and-separate self-supervised training procedure initially proposed by Zhao et al [57]. Instead of purely relying on visual semantic cues [57,17,19,53] or low-level optical-flow like motion representations [56], we consider to exploit the explicit human body and hand movements in the videos. To achieve this goal, we design a new framework, which consists of a video analysis network and an audio-visual separation network.…”

Audio-Visual Model for Generating Eating Sounds Using Food ASMR Videos

“…Gao et al [11] proposed a model to detect each musical instrument in a video clip of multiple sounds and divide the sound emitted from each instrument. Gan et al [10] improved the performance of time-frequency mask estimation for sound source separation using a context-aware graph network to extract information from the time series of the performer key points.…”

Improving Multimodal Speech Enhancement by Incorporating Self-Supervised and Curriculum Learning

“…It is instinctive to leverage the audio-visual synchronization as free supervision for training a neural network, which can recognize the objects that make sounds. Some works employ the learned self-supervised features to separate sound mixtures, including musical instrument sound separation [19,20,21,22] and speech separation [23], which do not need object/face/lip detection. Although these approaches show great promise that self-supervised learning assists to focus on the visual regions of sound sources, they may still fail to provide an acceptable solution in complicated auditory scenarios, e.g., the number of sound objects is uncertain and even dynamic.…”