What's All the FUSS About Free Universal Sound Separation Data?

Abstract

We introduce the Free Universal Sound Separation (FUSS) dataset, a new corpus for experiments in separating mixtures of an unknown number of sounds from an open domain of sound types. The dataset consists of 23 hours of single-source audio data drawn from 357 classes, which are used to create mixtures of one to four sources. To simulate reverberation, an acoustic room simulator is used to generate impulse responses of box shaped rooms with frequencydependent reflective walls. Additional open-source data augmentation tools are also provided to produce new mixtures with different combinations of sources and room simulations. Finally, we introduce an open-source baseline separation model, based on an improved time-domain convolutional network (TDCN++), that can separate a variable number of sources in a mixture. This model achieves 9.8 dB of scale-invariant signal-to-noise ratio improvement (SI-SNRi) on mixtures with two to four sources, while reconstructing single-source inputs with 35.5 dB absolute SI-SNR. We hope this dataset will lower the barrier to new research and allow for fast iteration and application of novel techniques from other machine learning domains to the sound separation challenge.

Compute and memory efficient universal sound source separation

“…Universal sound separation (variable number of sources 1-4): We also evaluate our models under a purely universal sound separation setup where multiple sound classes might be present and also we do not know how many sources are active in each input mixture. To that end, we use the FUSS benchmark dataset presented in [37]. FUSS contains sound clips that might contain at least one and up to four active sources per input mixture.…”

“…In order to be consistent with the state-of-the-art results on FUSS, we use the same dataset splits as the ones provided in [37]. The augmentation pipeline for each training mixture includes mixing sources from different training samples by sampling them uniformly over the batch.…”

“…N , but in reality the mixture might be comprised of only N < N active sources and N − N inactive sources. We follow a similar training procedure with the one presented in [37] and we force the network to produce zero outputs for the inactive slots after inferring the permutation that maximizes the total SNR. The loss for the active sources and inactive sources is defined in a permutation invariant sense as follows:…”

“…In order to evaluate the performance of our models we use a stable version of the permutation invariant SI-SDR improvement (SI-SDRi) as proposed in [37].…”

“…For the case where non active sources exist, thus, N < N , we omit to compute the aforementioned metric as it provides infinity values. However, we follow a stricter evaluation metric than the one proposed in [37] in the sense that we do not exclude pairs of estimates-targets with low energy estimated sources, as we believe that the evaluation metric should also reflect a penalty for the cases where the model under-separates the input mixture, leading to M < N non-zero estimated sources. For the variable number of sources case and specifically for the singlesource mixtures we simply report the maximum absolute SI-SDR obtained by each one of the estimated sources.…”