Improving Universal Sound Separation Using Sound Classification

Abstract

Deep learning approaches have recently achieved impressive performance on both audio source separation and sound classification. Most audio source separation approaches focus only on separating sources belonging to a restricted domain of source classes, such as speech and music. However, recent work has demonstrated the possibility of "universal sound separation", which aims to separate acoustic sources from an open domain, regardless of their class. In this paper, we utilize the semantic information learned by sound classifier networks trained on a vast amount of diverse sounds to improve universal sound separation. In particular, we show that semantic embeddings extracted from a sound classifier can be used to condition a separation network, providing it with useful additional information. This approach is especially useful in an iterative setup, where source estimates from an initial separation stage and their corresponding classifier-derived embeddings are fed to a second separation network. By performing a thorough hyperparameter search consisting of over a thousand experiments, we find that classifier embeddings from clean sources provide nearly one dB of SNR gain, and our best iterative models achieve a significant fraction of this oracle performance, establishing a new state-of-the-art for universal sound separation.

What’s all the Fuss about Free Universal Sound Separation Data?

“…Tzinis et al [4] performed separation experiments with a fixed number of sources on the 50-class ESC-50 dataset [5]. Other papers have leveraged information about sound class, either as conditioning information or as as a weak supervision signal [6,2,7].…”

“…Finally, we have released a masking-based baseline separation model, based on an improved time-domain convolutional network (TDCN++), described in our recent publications [1,2]. On the FUSS test set, this model achieves 9.8 dB of scale-invariant signal-to-noise ratio improvement (SI-SNRi) on mixtures with two to four sources, while reconstructing single-source inputs with 35.8 dB SI-SNR.…”

One-Shot Conditional Audio Filtering of Arbitrary Sounds

“…Kavalerov et al [1] have recently shown that by constructing a suitable dataset of mixtures of sounds, one can obtain SI-SDR (scale-invariant SDR) improvements of up to 10 dB. These results can be further improved by conditioning the sound separation model on the embeddings computed by a pre-trained audio classifier, which can also be fine-tuned [9]. More recently, a completely unsupervised approach to universal sound separation was developed [10], which does not require any single-source ground truth data, but instead can be trained using only real-world audio mixtures.…”

On Permutation Invariant Training For Speech Source Separation

“…The PASE loss term enforces the frame permutations to align with the best utterance permutation π * u . Previous works also show that conditioning source separation models with additional features improves performance [25][26][27], but whether feature conditioning helps reducing permutation errors has yet to be confirmed. Towards this end, we extend the single-stage uPIT+PASE paradigm to a two-stage cascaded system.…”