Alternative Objective Functions for Deep Clustering

This paper describes a dataset and protocols for evaluating continuous speech separation algorithms. Most prior studies on speech separation use pre-segmented signals of artificially mixed speech utterances which are mostly fully overlapped, and the algorithms are evaluated based on signal-to-distortion ratio or similar performance metrics. However, in natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. In addition, the signal-based metrics have very weak correlations with automatic speech recognition (ASR) accuracy. We think that not only does this make it hard to assess the practical relevance of the tested algorithms, it also hinders researchers from developing systems that can be readily applied to real scenarios. In this paper, we define continuous speech separation (CSS) as a task of generating a set of non-overlapped speech signals from a continuous audio stream that contains multiple utterances that are partially overlapped by a varying degree. A new real recorded dataset, called LibriCSS, is derived from LibriSpeech by concatenating the corpus utterances to simulate a conversation and capturing the audio replays with far-field microphones. A Kaldi-based ASR evaluation protocol is also established by using a well-trained multi-conditional acoustic model. By using this dataset, several aspects of a recently proposed speakerindependent CSS algorithm are investigated. The dataset and evaluation scripts are available to facilitate the research in this direction.

show abstract

“…permutation during model training. Numerous extentions to these methods were proposed with different focuses [3,4,5,6,7,8].…”

Section: Introductionmentioning

confidence: 99%

Continuous Speech Separation: Dataset and Analysis

Chen

Yoshioka

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

157

163

View full text Add to dashboard Cite

show abstract

“…In recent years multiple neural network based speech source separation techniques have been proposed. Techniques like deep clustering (DC) [1,2], deep attractor network (DAN) [3], u-PIT [4,5] and recurrent selective attention network (RSAN) [6] have achieved remarkable single-channel separation performance on mixtures of unknown speakers. All of the above mentioned separation systems rely on a transformation to the frequency domain, where separation masks are computed from the magnitude spectra.…”

Section: Introductionmentioning

confidence: 99%

Demystifying TasNet: A Dissecting Approach

Heitkaemper

Jakobeit

Boeddeker

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In recent years time domain speech separation has excelled over frequency domain separation in single channel scenarios and noise-free environments. In this paper we dissect the gains of the time-domain audio separation network (TasNet) approach by gradually replacing components of an utterance-level permutation invariant training (u-PIT) based separation system in the frequency domain until the TasNet system is reached, thus blending components of frequency domain approaches with those of time domain approaches. Some of the intermediate variants achieve comparable signal-to-distortion ratio (SDR) gains to TasNet, but retain the advantage of frequency domain processing: compatibility with classic signal processing tools such as frequency-domain beamforming and the human interpretability of the masks. Furthermore, we show that the scale invariant signalto-distortion ratio (si-SDR) criterion used as loss function in TasNet is related to a logarithmic mean square error criterion and that it is this criterion which contributes most reliable to the performance advantage of TasNet. Finally, we critically assess which gains in a noise-free single channel environment generalize to more realistic reverberant conditions.

show abstract

“…Loss Function Mask Type SDR Chimera++ [5] DC(W, WVA) tPSA 10.9 Our implementation DC(classic, WVA) tPSA 11.0 Table 1. Comparison between the chimera++ in [5] and our implementation…”

Section: Modelmentioning

confidence: 99%

“…Their results show that adding deep clustering leads to better mask predictions, thus achieving better separation performance. [5] made slight modifications to the original chimera network and developed several alternative loss functions for both deep clustering and mask-inference outputs. The new network (dubbed "chimera++") boosted separation performance using a much simpler architecture.…”

Section: Introductionmentioning

confidence: 99%

“…To estimate the phase of each separated source, [5] started with the estimated magnitude and the noisy phase and jointly reconstructed the phase of all sources using the multiple-input spectrogram inversion (MISI) algorithm [8]. Later, [9] unfolded the iterations in the MISI algorithm into separate STFT and iSTFT layers on top of a chimera++ network and trained the network to minimize the error in the time domain.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Mask-Dependent Phase Estimation for Monaural Speaker Separation

Mandel

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Speaker separation refers to isolating speech of interest in a multitalker environment. Most methods apply real-valued Time-Frequency (T-F) masks to the mixture Short-Time Fourier Transform (STFT) to reconstruct the clean speech. Hence there is an unavoidable mismatch between the phase of the reconstruction and the original phase of the clean speech. In this paper, we propose a simple yet effective phase estimation network that predicts the phase of the clean speech based on a T-F mask predicted by a chimera++ network. To overcome the label-permutation problem for both the T-F mask and the phase, we propose a mask-dependent permutation invariant training (PIT) criterion to select the phase signal based on the loss from the T-F mask prediction. We also propose an Inverse Mask Weighted Loss Function for phase prediction to focus the model on the T-F regions in which the phase is more difficult to predict. Results on the WSJ0-2mix dataset show that the phase estimation network achieves comparable performance to models that use iterative phase reconstruction or endto-end time-domain loss functions, but in a more straightforward manner.

show abstract

Alternative Objective Functions for Deep Clustering

Cited by 177 publications

References 8 publications

Continuous Speech Separation: Dataset and Analysis

Continuous Speech Separation: Dataset and Analysis

Demystifying TasNet: A Dissecting Approach

Mask-Dependent Phase Estimation for Monaural Speaker Separation

Contact Info

Product

Resources

About