Attention Is All You Need In Speech Separation

Abstract

Recurrent Neural Networks (RNNs) have long been the dominant architecture in sequence-to-sequence learning. RNNs, however, are inherently sequential models that do not allow parallelization of their computations. Transformers are emerging as a natural alternative to standard RNNs, replacing recurrent computations with a multi-head attention mechanism.In this paper, we propose the SepFormer, a novel RNN-free Transformer-based neural network for speech separation. The Sep-Former learns short and long-term dependencies with a multi-scale approach that employs transformers. The proposed model achieves state-of-the-art (SOTA) performance on the standard WSJ0-2/3mix datasets. It reaches an SI-SNRi of 22.3 dB on WSJ0-2mix and an SI-SNRi of 19.5 dB on WSJ0-3mix. The SepFormer inherits the parallelization advantages of Transformers and achieves a competitive performance even when downsampling the encoded representation by a factor of 8. It is thus significantly faster and it is less memory-demanding than the latest speech separation systems with comparable performance.

DF-Conformer: Integrated architecture of Conv-TasNet and Conformer using linear complexity self-attention for speech enhancement

“…From the results of experiments, we conclude that DF-Conformer is an effective model for SE. Future works include jointtraining of SE and ASR using an all Conformer model, and comparison with the dual-path methods [16][17][18] on the SE task.…”

“…One possible approach is to use the dual-path approach [16][17][18], which is equivalent to using sparse and block diagonal attention matrices corresponding to the inter-and intra-transformers, respectively. Alternatively, we use FAVOR+ attention introduced in Performer [26] which has linear computational complexity : O(N ).…”

“…Conv-TasNet [12] is a powerful model for SE that uses a combination of trainable analysis/synthesis filterbanks [13] and a mask prediction network using stacked 1-D dilated depthwise convolution (1D-DDC) layers. Since the denoising performance and computational efficiency are mainly affected by the mask prediction network, one of the main research topics in SE is improving the mask prediction architecture [14][15][16][17][18][19][20]. For example, the improved time-dilated convolution network (TDCN++) [14,15] extended Conv-TasNet to improve SE performance.…”

Separate But Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data

“…Recent advances in deep learning have enabled the development of neural network architectures capable of separating individual sound sources from mixtures of sounds with high fidelity. Discriminative separation models with supervised training have obtained stateof-the-art performance on multiple tasks such as music separation [1], speech separation [2,3] and speech enhancement [4,5]. However, gathering clean source waveforms to perform supervised training under various domains can be cumbersome or even impossible.…”

Dual-Path Filter Network: Speaker-Aware Modeling for Speech Separation

“…The main focus of Tas-Net is the separator that estimates the masks. A lot of work has since been done to improve the separator, such as fullyconvolutional TasNet (Conv-TasNet) [12], dual-path recurrent neural network (DPRNN) [19], gated DualPathRNN [14], dualpath transformer network (DPT-Net) [15], and SepFormer [20]. Among them, the dual-path method is the mainstream, which processes the waveform from the two dimensions of the local path and the global path.…”