What’s all the Fuss about Free Universal Sound Separation Data?

Abstract

We introduce the Free Universal Sound Separation (FUSS) dataset, a new corpus for experiments in separating mixtures of an unknown number of sounds from an open domain of sound types. The dataset consists of 23 hours of single-source audio data drawn from 357 classes, which are used to create mixtures of one to four sources. To simulate reverberation, an acoustic room simulator is used to generate impulse responses of box-shaped rooms with frequencydependent reflective walls. Additional open-source data augmentation tools are also provided to produce new mixtures with different combinations of sources and room simulations. Finally, we introduce an open-source baseline separation model, based on an improved time-domain convolutional network (TDCN++), that can separate a variable number of sources in a mixture. This model achieves 9.8 dB of scale-invariant signal-to-noise ratio improvement (SI-SNRi) on mixtures with two to four sources, while reconstructing single-source inputs with 35.8 dB absolute SI-SNR. We hope this dataset will lower the barrier to new research and allow for fast iteration and application of novel techniques from other machine learning domains to the sound separation challenge.

Sparse, Efficient, and Semantic Mixture Invariant Training: Taming In-the-Wild Unsupervised Sound Separation

“…We train separation networks using the same architecture as previous works [6,8,9,10], which separates sources by masking in a learned transform domain. The network is composed of a learnable encoder/decoder with 2.5 ms window and 1.25 ms hop, com-bined with a time-domain convolutional network (TDCN++).…”

“…Separation performance is evaluated using several supervised synthetic datasets containing reference sources. Since we train our separation models on a universal sound separation task, we primarily focus on evaluation with the FUSS dataset [9], which contains 10second mixtures of one to four arbitrary sound sources drawn from Since many practical separation applications focus on speech signals, we additionally evaluate how well unsupervised universal separation models can generalize to two specific task domains: speech separation, using mixtures of two overlapping speakers from Libr2Mix [22], and speech enhancement, using the same dataset as previous work [8] in which speech is drawn from librivox.org, and nonspeech from freesound.org.…”

“…We use several separation metrics, as proposed for measuring performance on FUSS [9] and for MixIT-trained models [8].…”

“…One outstanding issue with MixIT is that it can lead to models that over-separate, in the sense of producing a greater number of active source estimates than there are true underlying sources [8,9]. This is because the MixIT loss is invariant to the number of active estimated sources, as long as they can be remixed to approximate the reference mixtures with the same error.…”

Few-Shot Learning of New Sound Classes for Target Sound Extraction

“…These include AEs such as human sounds, object sounds, musical instruments, etc [10]. We generated two test sets using the FSD-Kaggle and a subset of the FSD50K data that consists of the AE sound samples from a single AE class provided in the FUSS dataset [24]. We used FSD50K to generate data with new AE classes unseen in the FSD-Kaggle training set.…”