Streaming Multi-Talker ASR with Token-Level Serialized Output Training

Abstract

This paper proposes a token-level serialized output training (t-SOT), a novel framework for streaming multi-talker automatic speech recognition (ASR). Unlike existing streaming multitalker ASR models using multiple output layers, the t-SOT model has only a single output layer that generates recognition tokens (e.g., words, subwords) of multiple speakers in chronological order based on their emission times. A special token that indicates the change of "virtual" output channels is introduced to keep track of the overlapping utterances. Compared to the prior streaming multi-talker ASR models, the t-SOT model has the advantages of less inference cost and a simpler model architecture. Moreover, in our experiments with LibriSpeechMix and LibriCSS datasets, the t-SOT-based transformer transducer model achieves the state-of-the-art word error rates by a significant margin to the prior results. For non-overlapping speech, the t-SOT model is on par with a single-talker ASR model in terms of both accuracy and computational cost, opening the door for deploying one model for both single-and multi-talker scenarios.

Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings

“…1. The t-SOT model was found to significantly outperform prior multitalker ASR models in both the recognition accuracy and latency while keeping the model architecture as simple as conventional single-talker ASR models [29].…”

“…The t-SOT framework was recently proposed to recognize multi-talker conversations with low latency [29]. In t-SOT, we assume up to M utterances are overlapping at the same time in the input audio.…”

“…Here, we explain the t-SOT for M = 2. Refer [29] for a case with M > 2. In t-SOT, the transcriptions for multiple speakers are serialized into a single sequence of recognition tokens (e.g., words, subwords) by sorting the tokens in a chronological order.…”

“…For the ASR block, we used a 18-layer or 36-layer TT (TT-18 or TT-36 in short) with the chunk-wise look-ahead proposed in [33] by using exactly the same configuration as in [29]. Each transformer block consisted of a 512-dim MHA with 8 heads and a 2048-dim point-wise feedforward layer.…”

“…To address these limitations, we propose a novel streaming SA-ASR model that works with low latency even for overlapping speech. Our model is based on token-level serialized output training (t-SOT) [29], which was recently proposed for low-latency multi-talker speech transcription. To further estimate the speaker identities, we propose an encoder-decoder based speaker embedding extractor that can estimate a speaker representation for each recognized token not only from nonoverlapping speech but also from overlapping speech.…”