Streaming End-to-End Multilingual Speech Recognition with Joint Language Identification

Abstract

Language identification is critical for many downstream tasks in automatic speech recognition (ASR), and is beneficial to integrate into multilingual end-to-end ASR as an additional task. In this paper, we propose to modify the structure of the cascadedencoder-based recurrent neural network transducer (RNN-T) model by integrating a per-frame language identifier (LID) predictor. RNN-T with cascaded encoders can achieve streaming ASR with low latency using first-pass decoding with no right-context, and achieve lower word error rates (WERs) using second-pass decoding with longer right-context. By leveraging such differences in the right-contexts and a streaming implementation of statistics pooling, the proposed method can achieve accurate streaming LID prediction with little extra testtime cost. Experimental results on a voice search dataset with 9 language locales shows that the proposed method achieves an average of 96.2% LID prediction accuracy and the same secondpass WER as that obtained by including oracle LID in the input.

Towards Zero-Shot Code-Switched Speech Recognition

“…( 6) of [31]; we denote this merged CTC likelihood as PCTC(Z|X). We then jointly decode PCTC(•) with an external bilingual LM, PB LM(Y ), using the time-synchronous beam search described in [43], which approximates the following decision: (12) where {V M ∪ V E } * denotes the set of all possible bilingual outputs. 3 This architecture, which we refer to as Conditional CTC, is 1 Unlike text-based transliteration [35], pseudo-labeling relies solely on the resources presumed to be available in our zero-shot CS ASR settings.…”

“…Therefore, a preeminent challenge in the CS ASR field is to build effective systems under the zero-shot setting where no CS ASR training data is available. Recent advancements in multilingual speech recognition have demonstrated the impressive scale of cross-lingual sharing in neural network approaches [3][4][5][6][7][8][9][10][11][12], and these works have shown that jointly modeling ASR with language identity (LID) grants some intra-sentential CS ability [11][12][13]. However, most of these large scale models skew towards highresourced languages [9] and do not seek to directly optimize for intra-sentential CS ASR between particular language pairs.…”

Improving Multilingual Transformer Transducer Models by Reducing Language Confusions

“…A neural transducer model [4] has three components: an acoustic encoder, a label prediction network, and a joint network. Neural transducer models can use different types of models as encoders such as LSTMs in RNN-T [4] and transformers [7,8,9,17,20,21,22] in transformer transducer (T-T). In this study, we use T-T as the backbone model for the development.…”

“…While end-to-end (E2E) models have made rapid progress in automatic speech recognition (ASR) [1][2][3][4][5][6][7][8], there are large amount of demands of multilingual ASR models since there are more than 60% people in the world can speak more than 2 languages according to [10]. There have been plenty of efforts to develop E2E multilingual models [11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26], and these models can achieve the comparable or even better ASR performance than monolingual baselines by passing the language identification (LID) information in the form of a one-hot or learnable embedding vector to distinguish different languages. In order to build streaming multilingual ASR systems for lots of practical applications that can perform similarly as the monolingual ones, we should not request users to input any LID information during model inference.…”

“…In order to build streaming multilingual ASR systems for lots of practical applications that can perform similarly as the monolingual ones, we should not request users to input any LID information during model inference. One solution is to infer LID as an embedding vector and attach it to the input features [18,19,21]. However, this kind of solution either leads to limited improvement due to LID prediction inaccuracy or introduces extra latency for reliable LID prediction [18,19].…”