Developing Real-Time Streaming Transformer Transducer for Speech Recognition on Large-Scale Dataset

Xie, Chen; Wu, Yu; Wang, Zhenghao; Liu, Shujie; Li, Jinyu

doi:10.1109/icassp39728.2021.9413535

Cited by 102 publications

(41 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the streaming ASR model, we used a TT with a chunkwise look-ahead proposed in [39]. The encoder consists of 2 convolution layers, each of which halves the time resolution, followed by the a 18-layer or 36-layer transformer with relative positional encoding.…”

Section: Experimental Settingsmentioning

confidence: 99%

See 1 more Smart Citation

Streaming Multi-Talker ASR with Token-Level Serialized Output Training

Kanda¹,

Wu²,

Wu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper proposes a token-level serialized output training (t-SOT), a novel framework for streaming multi-talker automatic speech recognition (ASR). Unlike existing streaming multitalker ASR models using multiple output layers, the t-SOT model has only a single output layer that generates recognition tokens (e.g., words, subwords) of multiple speakers in chronological order based on their emission times. A special token that indicates the change of "virtual" output channels is introduced to keep track of the overlapping utterances. Compared to the prior streaming multi-talker ASR models, the t-SOT model has the advantages of less inference cost and a simpler model architecture. Moreover, in our experiments with LibriSpeechMix and LibriCSS datasets, the t-SOT-based transformer transducer model achieves the state-of-the-art word error rates by a significant margin to the prior results. For non-overlapping speech, the t-SOT model is on par with a single-talker ASR model in terms of both accuracy and computational cost, opening the door for deploying one model for both single-and multi-talker scenarios.

show abstract

Section: Experimental Settingsmentioning

confidence: 99%

“…The audio input feature is an 80-dim log mel-filterbank extracted every 10 msec. As proposed in [39], we can control the algorithmic latency of We used an AdamW optimizer with a linear decay learning rate schedule with a peak learning rate of 1.5e-3 after 25K warm up iterations.…”

Section: Experimental Settingsmentioning

confidence: 99%

Streaming Multi-Talker ASR with Token-Level Serialized Output Training

Kanda¹,

Wu²,

Wu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The streaming mask function m(t) generates limited time frames [ts : te] ∈ [1 : T ] to compute the attention at time frame t by using an attention mask. With an appropriate attention mask, TT can be executed with limited algorithmic latency as proposed in [33]. Finally, FF * l () represents a position-wise feed forward network at the l-th layer.…”

Section: Transformer Transducermentioning

confidence: 99%

“…For the ASR block, we used a 18-layer or 36-layer TT (TT-18 or TT-36 in short) with the chunk-wise look-ahead proposed in [33] by using exactly the same configuration as in [29]. Each transformer block consisted of a 512-dim MHA with 8 heads and a 2048-dim point-wise feedforward layer.…”

Section: Experimental Settingsmentioning

confidence: 99%

Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings

Kanda¹,

Wu²,

Wu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents a streaming speaker-attributed automatic speech recognition (SA-ASR) model that can recognize "who spoke what" with low latency even when multiple people are speaking simultaneously. Our model is based on token-level serialized output training (t-SOT) which was recently proposed to transcribe multi-talker speech in a streaming fashion. To further recognize speaker identities, we propose an encoderdecoder based speaker embedding extractor that can estimate a speaker representation for each recognized token not only from non-overlapping speech but also from overlapping speech. The proposed speaker embedding, named t-vector, is extracted synchronously with the t-SOT ASR model, enabling joint execution of speaker identification (SID) or speaker diarization (SD) with the multi-talker transcription with low latency. We evaluate the proposed model for a joint task of ASR and SID/SD by using LibriSpeechMix and LibriCSS corpora. The proposed model achieves substantially better accuracy than a prior streaming model and shows comparable or sometimes even superior results to the state-of-the-art offline SA-ASR model.

show abstract

“…We employ Transformer Transducer model [20,21,22] as the backbone model structure. In the pre-training stage, we conduct the multi-task learning proposed in our previous work UniSpeech [16], where transducer loss and contrastive loss are combined together.…”

Section: Introductionmentioning

confidence: 99%

UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset

Wang,

Wu,

Liu

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, there has been a vast interest in self-supervised learning (SSL) where the model is pre-trained on large scale unlabeled data and then fine-tuned on a small labeled dataset. The common wisdom is that SSL helps resource-limited tasks in which only a limited amount of labeled data is available. The benefit of SSL keeps diminishing when the labeled training data amount increases. To our best knowledge, at most a few thousand hours of labeled data was used in the study of SSL. In contrast, the industry usually uses tens of thousands of hours of labeled data to build high-accuracy speech recognition (ASR) systems for resource-rich languages. In this study, we take the challenge to investigate whether and how SSL can improve the ASR accuracy of a state-of-the-art production-scale Transformer-Transducer model, which was built with 65 thousand hours of anonymized labeled EN-US data.

show abstract

Developing Real-Time Streaming Transformer Transducer for Speech Recognition on Large-Scale Dataset

Cited by 102 publications

References 35 publications

Streaming Multi-Talker ASR with Token-Level Serialized Output Training

Streaming Multi-Talker ASR with Token-Level Serialized Output Training

Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings

UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset

Contact Info

Product

Resources

About