On-the-Fly Audio Source Separation—A Novel User-Friendly Framework

Abstract

Abstract-This article addresses the challenging problem of single-channel audio source separation. We introduce a novel user-guided framework where source models that govern the separation process are learned on-the-fly from audio examples retrieved online. The user only provides the search keywords that describe the sources in the mixture. In this framework, the generic spectral characteristics of each source are modeled by a universal sound class model learned from the retrieved examples via non-negative matrix factorization. We propose several group sparsity-inducing constraints in order to efficiently exploit a relevant subset of the universal model adapted to the mixture to be separated. We then derive the corresponding multiplicative update rules for parameter estimation. Separation results obtained from automated and user tests on mixtures containing various types of sounds confirm the effectiveness of the proposed framework.Index Terms-On-the-fly audio source separation, user-guided, non-negative matrix factorization, group sparsity, universal sound class model.

Gaussian Modeling-Based Multichannel Audio Source Separation Exploiting Generic Source Spectral Model

“…The number of NMF components in W l j for each speech example was set to 32, while that for noise example was 16. These values were found to be reasonable in [15] and our work on single-channel case [18]. Each W l j were obtained by optimizing (17) with 20 MU iterations.…”

“…. This leads to a straightforward extension of the conventional optimization criterion described by (15) where H j is now estimated by optimizing the criterion:…”

“…Following this trend, some recent works including ours have proposed to use a very abstract semantic information just about the types of audio sources existing in the mixture to guide the source separation. If one source in the mixture is known as "speech", then several speaker-independent speech examples can be used to create a universal speech model as presented in [14]; if several types of sound sources in the mixture are known (e.g., birdsong, piano, waterfall), their audio examples found by internet search can be used to learn the corresponding universal sound class models as presented in [15]. Such universal models were shown to be effective in guiding the source separation algorithm and resulted in promising performance.…”

“…In this paper, we present an extension of the previous works [15], [16], [18] to the multichannel case where the NMF-based GSSM is combined with the full-rank spatial covariance model in a Gaussian modeling paradigm. Around this LGM, existing works have investigated several source spectral models such as Gaussian mixture model (GMM) [37], NMF as a linear model with nonnegativity constraints [36], [38], continuity model [39], kernel additive model [40], heavy-tailed distributionsbased model [41], [42], and recently DNN [24].…”

Class-conditional Embeddings for Music Source Separation

“…We also explore what type of covariance model is most effective for musical source separation (tied vs. untied across classes, diagonal vs. spherical). Furthermore, we discuss a simple modification of our pre-trained embedding networks for query-by-example separation [18,19,20], where given an isolated example of a sound we want to separate, we can extract the portion of a mixture most like the query without supervision.…”