Exploring Retraining-free Speech Recognition for Intra-sentential Code-switching

Abstract

In this paper, we present our initial efforts for building a codeswitching (CS) speech recognition system leveraging existing acoustic models (AMs) and language models (LMs), i.e., no training required, and specifically targeting intra-sentential switching. To achieve such an ambitious goal, new mechanisms for foreign pronunciation generation and language model (LM) enrichment have been devised. Specifically, we have designed an automatic approach to obtain high quality pronunciation of foreign language (FL) words in the native language (NL) phoneme set using existing acoustic phone decoders and an LSTM-based grapheme-to-phoneme (G2P) model. Improved accented pronunciations have thus been obtained by learning foreign pronunciations directly from data. Furthermore, a codeswitching LM was deployed by converting the original NL LM into a CS LM using translated word pairs and borrowing statistics for the NL LM. Experimental evidence clearly demonstrates that our approach better deals with accented foreign pronunciations than techniques based on human labeling. Moreover, our best system achieves a 55.5% relative word error rate reduction from 34.4%, obtained with a conventional monolingual ASR system, to 15.3% on an intra-sentential CS task without harming the monolingual recognition accuracy.

Transformer-Transducers for Code-Switched Speech Recognition

“…They looked at phone-merging techniques to handle the two languages in acoustic modeling, explored further in [9,31], and generating codeswitched text data for language modeling, studied more in [32,39]. Since then, different approaches have been applied to improve codeswitched speech recognition like speech chains [40], transliteration [41], and translation [42]. Authors in [14,27,43] focus on tracking the language switch points, similar to our LID aware training.…”

Code-Switch Speech Rescoring with Monolingual Data

“…We also apply the multi-graph decoding strategy [17], which contains code-switch and English 1 and the code-switch dataset as Table 2, and the English is trained by the transcript of librispeech. Besides, we implement the methods that optimizes the NGram language model in [7] and [18] as the baseline of the first-pass decoding, and these two methods together achieve 7% relative WER reduction. The number of NBEST is set to 128.…”