A Style Transfer Approach to Source Separation

Abstract

Training neural networks for source separation involves presenting a mixture recording at the input of the network and updating network parameters in order to produce an output that resembles the clean source. Consequently, supervised source separation depends on the availability of paired mixture-clean training examples. In this paper, we interpret source separation as a style transfer problem. We present a variational auto-encoder network that exploits the commonality across the domain of mixtures and the domain of clean sounds and learns a shared latent representation across the two domains. Using these cycle-consistent variational auto-encoders, we learn a mapping from the mixture domain to the domain of clean sounds and perform source separation without explicitly supervising with paired training examples.

Content Based Singing Voice Extraction from a Musical Mixture

“…In music, the most useful application is that of separating the lead vocals from a musical mixture. This problem is well researched and numerous deep learning based models have recently been proposed to tackle it [4,5,6,7,8,9,10,11]. Most of these models use the neural network to predict soft time frequency masks, given an input magnitude spectrogram of the mixture signal.…”

Self-supervised Learning for Speech Enhancement

“…To relax the constraints of paired training data, a few recent approaches interpret the problem of denoising and source separation as a style-transfer problem wherein, the goal is to map from the domain of noisy mixtures to the domain of clean sounds (Stoller et al, 2018;Michelashvili et al, 2019;Venkataramani et al, 2019). These approaches only require a training set of mixtures and a training set of clean sounds, but the clean sounds can be unpaired and unrelated to the mixtures.…”