Joint ASR and Language Identification Using RNN-T: An Efficient Approach to Dynamic Language Switching

Abstract

Conventional dynamic language switching enables seamless multilingual interactions by running several monolingual ASR systems in parallel and triggering the appropriate downstream components using a standalone language identification (LID) service. Since this solution is neither scalable nor cost-and memory-efficient, especially for on-device applications, we propose end-to-end, streaming, joint ASR-LID architectures based on the recurrent neural network transducer framework. Two key formulations are explored: (1) joint training using a unified output space for ASR and LID vocabularies, and (2) joint training viewed as multi-task optimization. We also evaluate the benefit of using auxiliary language information obtained on-thefly from an acoustic LID classifier. Experiments with the English-Hindi language pair show that: (a) multi-task architectures perform better overall, and (b) the best joint architecture surpasses monolingual ASR (6.4-9.2% word error rate reduction) and acoustic LID (53.9-56.1% error rate reduction) baselines while reducing the overall memory footprint by up to 46%.

Streaming End-to-End Multilingual Speech Recognition with Joint Language Identification

“…x 1:t < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 2 g 5 G J 3 It is known that acoustic and linguistic information can be combined to improve LID prediction [9,12,28]. The concatenation of e enc1 r:t+r and e enc2 1:t allows the LID predictor to leverage such complementary information easily.…”

“…Our work differs from this as we predict the LIDs instead of using the oracle ones. Another body of work looks at techniques to predict LID and use the predictions in the ASR system or downstream tasks [21][22][23][24][25][26][27][28][29]. Much of this work focuses on LID predictions in a non-streaming system [23][24][25][26][30][31][32], which does not fit into our streaming ASR setup that is important due to production constraints.…”

“…Much of this work focuses on LID predictions in a non-streaming system [23][24][25][26][30][31][32], which does not fit into our streaming ASR setup that is important due to production constraints. Our work also differs from existing streaming work [27][28][29], as we use an RNN-T cascaded encoder model with a 0.9-second delay in the second-pass, making it suitable for accurate LID predictions. Furthermore, our LID predictor is lightweight as it uses non-parametric streaming statistics pooling and increases parameters by only 0.5%.…”

“…Furthermore, our LID predictor is lightweight as it uses non-parametric streaming statistics pooling and increases parameters by only 0.5%. Finally, compared to previous works, our experiments were conducted on a largerscale of 9 language locales [27][28][29].…”

Bilingual End-to-End ASR with Byte-Level Subwords

“…The resulting E2E model can perform utterance-based multilingual ASR. The works in [4] [5] [6] [7] aim to build an E2E model that can improve code switching. While these approaches are different from each other, there are some similarities among them.…”