The Sound of Motions

Abstract

Sounds originate from object motions and vibrations of surrounding air. Inspired by the fact that humans is capable of interpreting sound sources from how objects move visually, we propose a novel system that explicitly captures such motion cues for the task of sound localization and separation. Our system is composed of an end-to-end learnable model called Deep Dense Trajectory (DDT), and a curriculum learning scheme. It exploits the inherent coherence of audio-visual signals from a large quantities of unlabeled videos. Quantitative and qualitative evaluations show that comparing to previous models that rely on visual appearance cues, our motion based system improves performance in separating musical instrument sounds. Furthermore, it separates sound components from duets of the same category of instruments, a challenging problem that has not been addressed before.

Music Gesture for Visual Sound Separation

“…Most recently, Zhao et al [56] proposed to leverage temporal motion information to improve the vision sound separation. However, this algorithm has not yet seen wide applicability to sound separation on real mixtures.…”

“…We perform experiments on three video music performance datasets, namely MUSIC-21 [56], URMP [31] and AtinPiano [36]. MUSIC-21 is an untrimmed video dataset crawled by keyword query from Youtube.…”

“…Our model is built on the mix-and-separate self-supervised training procedure initially proposed by Zhao et al [57]. Instead of purely relying on visual semantic cues [57,17,19,53] or low-level optical-flow like motion representations [56], we consider to exploit the explicit human body and hand movements in the videos. To achieve this goal, we design a new framework, which consists of a video analysis network and an audio-visual separation network.…”

Self-Supervised Moving Vehicle Tracking With Stereo Sound

“…[18,26] also explored how to generate spatial sound for videos. More recently, [40,39,13,17,32] used the visual-audio correspondence to separate sound sources. In contrast to previous work that has only transferred class-level information between modalities, this work transfers richer, region-level location information about objects.…”

Foley Music: Learning to Generate Music from Videos

“…For example, the associations between speech and facial movements can be used for facial animations from speech [31,55], generating high-quality talking face from audio [54,30], separate mixed speech signals of multiple speakers [14,42], and even lip-reading from raw videos [12]. Zhao et al [63] and Zhou et al [68] have demonstrated to use optical flow like motion representations to improve the quality of visual sound separations and sound generations. There are also some recent works to explore the correlations between body motion and sound by predicting gestures from speech [22], body dynamics from music [50], or identifying a melody through body language [15].…”