Wavesplit: End-to-End Speech Separation by Speaker Clustering

Abstract

We introduce Wavesplit, an end-to-end speech separation system. From a single recording of mixed speech, the model infers and clusters representations of each speaker and then estimates each source signal conditioned on the inferred representations. The model is trained on the raw waveform to jointly perform the two tasks. Our model infers a set of speaker representations through clustering, which addresses the fundamental permutation problem of speech separation. Moreover, the sequence-wide speaker representations provide a more robust separation of long, challenging sequences, compared to previous approaches. We show that Wavesplit outperforms the previous state-of-the-art on clean mixtures of 2 or 3 speakers (WSJ0-2mix, WSJ0-3mix), as well as in noisy (WHAM!) and reverberated (WHAMR!) conditions. As an additional contribution, we further improve our model by introducing online data augmentation for separation.

Stepwise-Refining Speech Separation Network via Fine-Grained Encoding in High-order Latent Domain

“…Table II presents the experimental results of different models for speech separation on WSJ0-2mix dataset [39] in terms of both ∆SI-SNR and ∆SDR. For the results without using data augmentation, our DPRNN-SRSSN outperforms all other methods except Wavesplit [20] in terms of both metrics while DPTNET-SRSSN performs better than all other methods, which demonstrates advantages of our model. The methods which learn a separable encoding space defined by a latent domain, generally perform better than the other type of methods separating speech in frequency domain explicitly, which implies that the frequency domain is not necessarily the best separation space for speech as described in [10].…”

“…2) Comparison with State-of-the-art Methods on WSJ0-2mix (involving 2 speakers): Next we conduct experiments to compare our model with state-of-the-art methods for speech separation on WSJ0-2mix dataset [2]. In particular, we compare our model with 2 types of methods: 1) methods performing separation in the frequency domain, including DPCL++ [3], UPIT-Bi-LSTM-ST [5], Chimera++ [7] and Deep CASA [8]; 2) methods performing separation in a learnable latent domain in an end-to-end way, including Bi-LSTM-TASNET [12], Conv-TASNET [10], E2EPF [28], FurcaNeXt [13], DRPNN-TASNET [15], SuDoRM-RF [17], Nachmani et al [16], DPTNET-TASNET [18], SepFormer [19] and Wavesplit [20]. We evaluate the performance of two versions of our SRSSN: DPRNN-SRSSN and DPTNET-SRSSN.…”

“…With great success of deep learning in many fields such as computer vision and machine learning by the powerful capability of feature representation, another research line of speech separation [10]- [20] seeks to leverage deep convolutional neural networks (CNNs) to learn an embedding space that is separable for speech signals between different speakers. A remarkable benefit of such methods is that the whole separation procedures including encoding, separation and decoding can be integrated into an end-to-end model, which is in contrast to the sequentially individual steps in aforementioned STFT-based methods.…”

“…Nachmani et al [16] introduce a task to minimize the distance between the speaker embeddings of the estimated signal and target signal, where the speaker embeddings are extracted from a separately trained speaker identification network. Zeghidour et al [20] propose the Wavesplit network consisting of a speaker stack and a separation stack, where the speaker stack extracts frame-level speaker-discriminative vectors and obtain speaker centroids employing clustering, and the separation stack estimates isolated speech signals conditioned the speaker centroids. Two-stage or cascaded architectures.…”

A Conformer-based ASR Frontend for Joint Acoustic Echo Cancellation, Speech Enhancement and Speech Separation

“…When using speaker embedding, the target speaker of interest is assumed to be known a priori. Techniques developed for speaker separation have also been applied to remove nonspeech noise [32,33], with modifications to the training data. AEC has also been studied in isolation [34], or together with background noise [35].…”