Towards Language-Universal Mandarin-English Speech Recognition

Abstract

Multilingual and code-switching speech recognition are two challenging tasks that are studied separately in many previous works. In this work, we jointly study multilingual and codeswitching problems, and present a language-universal bilingual system for Mandarin-English speech recognition. Specifically, we propose a novel bilingual acoustic model, which consists of two monolingual system initialized subnets and a shared output layer corresponding to the Character-Subword acoustic modeling units. The bilingual acoustic model is trained using a large Mandarin-English corpus with CTC and sMBR criteria. We find that this model, which is not given any information about language identity, can achieve comparable performance in monolingual Mandarin and English test sets compared to the well-trained language-specific Mandarin and English ASR systems, respectively. More importantly, the proposed bilingual model can automatically learn the language switching. Experimental results on a Mandarin-English code-switching test set show that it can achieve 11.8% and 17.9% relative error reduction on Mandarin and English parts, respectively.

ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems

“…Similarly, a phoneme-based modeling unit was also studied to achieve the multilingual ASR task [44]. A network sharing approach was also developed to recognize the Chinese and English languages [45]. A multi-task learning mechanism was proposed to obtain an end-to-end multilingual task in [46].…”

Decoupling recognition and transcription in Mandarin ASR

“…Their results showed that bytes are superior to grapheme characters over a wide variety of languages in monolingual end-to-end speech recognition. Characters are the most commonly used modeling unit for end-to-end ASR in Mandarin Chinese; sub-words have also been employed [45].…”

Language-specific Characteristic Assistance for Code-switching Speech Recognition

“…Compared to monolingual ASR with plenty of monolingual data, CS ASR is limited by hard-to-collect speech and transcriptions, especially in the era of deep learning. Therefore, reducing the demand for CS data and making full use of rich resources monolingual data have become research hotspots [13,14,15,16,17,18]. Dual-encoder structure is an effective way to make full use of two monolingual data [15,16,17,18].…”