Streaming Multi-Speaker ASR with RNN-T

Abstract

Recent research shows end-to-end ASR systems can recognize overlapped speech from multiple speakers. However, all published works have assumed no latency constraints during inference, which does not hold for most voice assistant interactions. This work focuses on multi-speaker speech recognition based on a recurrent neural network transducer (RNN-T) that has been shown to provide high recognition accuracy at a low latency online recognition regime. We investigate two approaches to multi-speaker model training of the RNN-T: deterministic output-target assignment and permutation invariant training. We show that guiding separation with speaker order labels in the former case enhances the high-level speaker tracking capability of RNN-T. Apart from that, with multistyle training on single-and multi-speaker utterances, the resulting models gain robustness against ambiguous numbers of speakers during inference. Our best model achieves a WER of 10.2% on simulated 2-speaker LibriSpeech data, which is competitive with the previously reported state-of-the-art nonstreaming model (10.3%), while the proposed model could be directly applied for streaming applications.

Streaming Multi-Talker ASR with Token-Level Serialized Output Training

“…Firstly, we observed that the t-SOT TT-18 with only 40 msec algorithmic latency already outperformed the results of all prior streaming multi-talker ASR models. Note that even though t-SOT TT-18 has almost the same number of parameters with SURT [26,32] or MS-RNN-T [27,34], t-SOT is more space and computationally efficient in the inference because SURT and MS-RNN-T run decoding twice, once for each of the two output branches. Secondly, we observed a significant WER reduction by increasing algorithmic latency and the model size.…”

“…For streaming multi-talker ASR, the t-SOT framework has various advantages over SURT [26] and MS-RNN-T [27]. Firstly, t-SOT requires only a single decoding process as with the conventional single-talker ASR while SURT and MS-RNN-T require to execute the decoder multiple times (i.e., one decoder run for each output branches).…”

“…A few recent studies explored the streaming multi-talker ASR problem to transcribe each spoken word with a low latency even for overlapping speech. Streaming unmixing and recognition transducer (SURT) [26] and multi-speaker recurrent neural network transducer (MS-RNN-T) [27] were concurrently proposed based on a similar idea, where the model has two output branches to generate two simultaneous transcriptions for overlapping speech. However, their reported WERs still lagged far behind the SOTA result of the offline SOT-model.…”

“…A special token indicating "virtual" output channels is used to keep track of the transcriptions for different speakers. Compared to the prior streaming multi-talker ASR models [26,27], the t-SOT model has the advantages of less inference cost and a simpler model architecture. Moreover, our experimental results with LibriSpeechMix and LibriCSS datasets show that the transformer transducer (TT) [28] trained with t-SOT framework achieves a new SOTA results by a significant margin to the prior results including the offline models.…”

A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio

“…While promising results were shown for such joint systems, most of the previous studies were limited to either simulated data [15,17,25,[28][29][30][31][32][33][34][35] or small-scale real data [11,[36][37][38]. It is because of the scarcity of training data for real meeting recordings, which takes a lot of time to precisely transcribe.…”