End-To-End Multi-Speaker Speech Recognition With Transformer

Abstract

Recently, fully recurrent neural network (RNN) based endto-end models have been proven to be effective for multi-speaker speech recognition in both the single-channel and multi-channel scenarios. In this work, we explore the use of Transformer models for these tasks by focusing on two aspects. First, we replace the RNN-based encoder-decoder in the speech recognition model with a Transformer architecture. Second, in order to use the Transformer in the masking network of the neural beamformer in the multi-channel case, we modify the self-attention component to be restricted to a segment rather than the whole sequence in order to reduce computation. Besides the model architecture improvements, we also incorporate an external dereverberation preprocessing, the weighted prediction error (WPE), enabling our model to handle reverberated signals. Experiments on the spatialized wsj1-2mix corpus show that the Transformer-based models achieve 40.9% and 25.6% relative WER reduction, down to 12.1% and 6.4% WER, under the anechoic condition in single-channel and multi-channel tasks, respectively, while in the reverberant case, our methods achieve 41.5% and 13.8% relative WER reduction, down to 16.5% and 15.2% WER. Attention-DecoderEnc 1 SD < l a t e x i t s h a 1 _ b a s e 6 4 = " Z E x 5 4 u 5 D h 9 n r k V q v e X 1 a T 1 D L f 2 g = " > A A A C A X i c b V D L S s N A F J 3 U V 6 2 v q B v B T b A I r k p S B V 0 W V H B Z 0 T 6 g j W E y n b R D J 5 M w c y O W E D f + i h s X i r j 1 L 9 z 5 N 0 7 T L r T 1 w M D h n H u 5 c 4 4 f c 6 b A t r + N w s L i 0 v J K c b W 0 t r 6 x u W V u 7 < l a t e x i t s h a 1 _ b a s e 6 4 = " / 1 B 3 S g + + 4 5 a r q j n e 5 e M f V V 5 b I N Y = " > A A A C A X i c b V D L S s N A F J 3 U V 6 2 v q B v B T b A I r k p S B V 0 W V H B Z 0 T 6 g j W E y n b R D J 5 M w c y O W E D f + i h s X i r j 1 L 9 z 5 N 0 7 T L r T 1 w M D h n H u 5 c 4 4 f c 6 b A t r + N w s L i 0 v J K c b W 0 t r 6 x u W V u 7 B o m w M L K m b V h 9 L o G h m G h C m e T 6 r x Y b U U k Z 6 s 7 K u g R n P v I i a d W q z l m 1 d n t e q d t F H S V y R I 7 J K X H I B a m T G 9 I g T c J I R p 7 J K 3 k z n o w X 4 9 3 4 m I 0 u G c X O A f k D 4 / M H + Y S X P A = = < / l a t e x i t > EncMix < l a t e x i t s h a 1 _ b a s e 6 4 = " t H P Y 8 W F E w 4 g s 1 l 5 Q M 1 a 7 j E W l J 1 Y = " > A A A C A H i c b V D L S s N A F J 3 U V 6 2 v q A s X b g a L 4 K o k V d B l Q Q Q 3 Q g X 7 g D a E y A 2 T 4 V C s D 7 E d S v x B w r v 7 e S E m g 1 D T w 9 G R A Y K Q W v Z n 4 n 9 d L w L 9 y U h 7 G C T A d M T / k J w J D h G d t 4 A G X j I K Y a k K o 5 P q v m I 6 I J B R 0 Z x V d g r 0 Y e Z m 0 6 z X 7 v F a / v 6 g 2 r K K O M j p G J + g M 2 e g S N d A t a q I W o i h D z + g V v R l P x o v x b n z M R 0 t G s X O I / s D 4 / A

Gated Recurrent Fusion With Joint Training Framework for Robust End-to-End Speech Recognition

“…The third mainstream method is the joint training methods [21], [26]- [28]. These methods apply the joint training framework to optimize the speech enhancement and recognition, simultaneously.…”

End-to-End Multi-Channel Transformer for Speech Recognition

“…One approach to make these devices robust against noise is to equip them with multiple microphones so that the spectral and spatial diversity of the target and interference signals can be leveraged using beamforming approaches [1][2][3][4][5][6]. It has been demonstrated in [4,6,7] that beamforming methods for multi-channel speech enhancement produce substantial improvements for ASR systems; therefore, existing ASR pipelines are mainly built on beamforming as a pre-processor and then cascaded with an acoustic-to-text model [2,[8][9][10].…”

Audio-Visual Multi-Channel Recognition of Overlapped Speech

“…The neural Filter&Sum approaches directly estimate the beamforming filter parameters in either time domain [16]- [18] or frequency domain [19] to produce the separated outputs. The mask-based MVDR [4]- [6], [20]- [23] and related mask-based GEV [24], [25] approaches predict the TF masks using DNNs before estimating the power spectral density (PSD) matrices for the target and overlapping speakers to obtain the beamforming filter parameters. Compared with the conventional stand-alone beamforming approaches, these neural based methods allow a arXiv:2011.07755v1 [eess.AS] 16 Nov 2020 tighter integration with the downstream recognition back-end [5], [6], [19], [25], [26].…”

“…The mask-based MVDR [4]- [6], [20]- [23] and related mask-based GEV [24], [25] approaches predict the TF masks using DNNs before estimating the power spectral density (PSD) matrices for the target and overlapping speakers to obtain the beamforming filter parameters. Compared with the conventional stand-alone beamforming approaches, these neural based methods allow a arXiv:2011.07755v1 [eess.AS] 16 Nov 2020 tighter integration with the downstream recognition back-end [5], [6], [19], [25], [26]. Large performance improvements have been reported for overlapped speech recognition tasks by using microphone array based multi-channel inputs [5], [6].…”

“…Compared with the conventional stand-alone beamforming approaches, these neural based methods allow a arXiv:2011.07755v1 [eess.AS] 16 Nov 2020 tighter integration with the downstream recognition back-end [5], [6], [19], [25], [26]. Large performance improvements have been reported for overlapped speech recognition tasks by using microphone array based multi-channel inputs [5], [6]. However, the current systems' performance gap between overlapped and non-overlapped speech remains large.…”