End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction

Abstract

This paper proposes an end-to-end approach for single-channel speaker-independent multi-speaker speech separation, where time-frequency (T-F) masking, the short-time Fourier transform (STFT), and its inverse are represented as layers within a deep network. Previous approaches, rather than computing a loss on the reconstructed signal, used a surrogate loss based on the target STFT magnitudes. This ignores reconstruction error introduced by phase inconsistency. In our approach, the loss function is directly defined on the reconstructed signals, which are optimized for best separation. In addition, we train through unfolded iterations of a phase reconstruction algorithm, represented as a series of STFT and inverse STFT layers. While mask values are typically limited to lie between zero and one for approaches using the mixture phase for reconstruction, this limitation is less relevant if the estimated magnitudes are to be used together with phase reconstruction. We thus propose several novel activation functions for the output layer of the T-F masking, to allow mask values beyond one. On the publiclyavailable wsj0-2mix dataset, our approach achieves state-ofthe-art 12.6 dB scale-invariant signal-to-distortion ratio (SI-SDR) and 13.1 dB SDR, revealing new possibilities for deep learning based phase reconstruction and representing a fundamental progress towards solving the notoriously-hard cocktail party problem.

Multi-Microphone Complex Spectral Mapping for Speech Dereverberation

“…A major difference from [31], [33] is that we do not perform power or logarithmic compression on the magnitude spectra. This way, the DNN is always trained to estimate an STFT spectrogram that has consistent phase and magnitude structure, and hence would likely produce a good consistent STFT spectrogram at run time [34], [35].…”

Phase Reconstruction Based On Recurrent Phase Unwrapping With Deep Neural Networks

“…Motivated by the recent advance in deep learning, several DNNbased phase reconstruction methods have been presented [18][19][20][21][22][23]. However, phase reconstruction from a given amplitude spectrogram is not an easy task for DNNs due to the following two problems: the wrapping effect and sensitivity to a shift of a waveform.…”

CLCNET: Deep Learning-Based Noise Reduction for Hearing aids using Complex Linear Coding

“…Removing unwanted environmental background noise in speech signals is a common step in speech processing applications. Complex valued neural networks as well as phase estimation have been of great interest in speech enhancement lately, since the perceptual audio quality has been reported to be improved significantly [5,7,6,10].…”

“…While other work uses the whole signal in an off-line processing fashion as input for the noise reduction [9,10,11], our work requires real-time capabilities. Both high-res spectrograms and off-line processing are not feasible for hearing aid applications, where the overall latency is a very important property.…”