Separate But Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data

Abstract

We propose FEDENHANCE, an unsupervised federated learning (FL) approach for speech enhancement and separation with non-IID distributed data across multiple clients. We simulate a realworld scenario where each client only has access to a few noisy recordings from a limited and disjoint number of speakers (hence non-IID). Each client trains their model in isolation using mixture invariant training while periodically providing updates to a central server. Our experiments show that our approach achieves competitive enhancement performance compared to IID training on a single device and that we can further facilitate the convergence speed and the overall performance using transfer learning on the server-side. Moreover, we show that we can effectively combine updates from clients trained locally with supervised and unsupervised losses. We also release a new dataset LibriFSD50K and its creation recipe in order to facilitate FL research for source separation problems.

RemixIT: Continual Self-Training of Speech Enhancement Models via Bootstrapped Remixing

“…If the noise sources are independent from each other and the clean speech component, then the model can learn to minimize this loss by reconstructing the mixture using its first estimated slot and either one of the two noise slots available. Although MixIT has been proven effective for various simulated speech enhancement setups [29], [30], the assumption about having access to a diverse set of in-domain noise recordings from D n which aptly captures the true distribution of the present background noises D * n make it impractical for many real-world settings. To this end, other works [24], [31] have tried to deal with the distribution shift between the on-hand noise dataset D n and the actual noise distribution D * n in order to avoid the need of in-domain noise samples.…”

“…The clean speech samples are drawn from the LibriSpeech [51] corpus and the noise recordings are taken from FSD50K [52] representing a set of almost 200 classes of background noises after excluding all the human-made sounds from the AudioSet ontology [53]. A detailed recipe of the dataset generation process is presented in [30]. LFSD becomes an ideal candidate for semi-supervised/SSL teacher pre-training on OOD data given its mixture diversity.…”

“…: The generation process for this dataset produces 20,000 training noisy-speech pairs and 3,000 test mixtures from the initial WHAM! [54] dataset and has been identical to the procedure followed in [30] with active noise sources mixed at an average of −1.3dB input SNR. The set of background noises in WHAM!…”

“…Mixture invariant training (MixIT) [28] enables unsupervised training of separation models only with real-world singlechannel recordings by generating artificial mixtures of mixtures and estimating the independent sources. Although MixIT has been proven successful for various speech enhancement tasks [28], [29], [30], MixIT assumes access to in-domain noise samples which restricts its universal applicability. Overcoming the latter constraint by injecting additional out-of-domain (OOD) noise sources to the input mixture of mixtures [31] further alters the input signal-to-noise ratio (SNR) distribution and its performance depends heavily on the distribution shift between the injected and real noise distributions.…”

STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency

“…We sample the clips in the development set of FSD50k to simulate the noises for training and validation, and those in the evaluation set to simulate the noises for testing. Since our task is single-speaker speech enhancement, following [50] we filter out clips containing any sounds produced by humans, based on the provided sound event annotation of each clip. Such clips have annotations such as Human voice, Male speech and man speaking, Chuckle and chortle, Yell, etc 2 .…”