Multi-Graph Decoding for Code-Switching ASR

Abstract

In the FAME! Project, a code-switching (CS) automatic speech recognition (ASR) system for Frisian-Dutch speech is developed that can accurately transcribe the local broadcaster's bilingual archives with CS speech. This archive contains recordings with monolingual Frisian and Dutch speech segments as well as Frisian-Dutch CS speech, hence the recognition performance on monolingual segments is also vital for accurate transcriptions. In this work, we propose a multi-graph decoding and rescoring strategy using bilingual and monolingual graphs together with a unified acoustic model for CS ASR. The proposed decoding scheme gives the freedom to design and employ alternative search spaces for each (monolingual or bilingual) recognition task and enables the effective use of monolingual resources of the high-resourced mixed language in low-resourced CS scenarios. In our scenario, Dutch is the high-resourced and Frisian is the low-resourced language. We therefore use additional monolingual Dutch text resources to improve the Dutch language model (LM) and compare the performance of single-and multi-graph CS ASR systems on Dutch segments using larger Dutch LMs. The ASR results show that the proposed approach outperforms baseline single-graph CS ASR systems, providing better performance on the monolingual Dutch segments without any accuracy loss on monolingual Frisian and code-mixed segments.

End-to-End Code-Switching ASR for Low-Resourced Language Pairs

“…Finally, the ASR results provided by the multi-graph E2E ASR systems are presented in the bottom panel. According to these results, using an additional monolingual Frisian graph during the multi-graph decoding (union-fy and union-fy-nl) does not improve the ASR performance on the fy utterances, which is consistent with the previous results reported in [26]. Including the largest monolingual Dutch graph in the unionfy-nl++ system improves the ASR accuracy on nl utterances with a WER of 30.1% (25.6%), yielding a 10.7% (11.7%) relative WER reduction.…”

“…In our previous work [26], Yilmaz et al proposed a multigraph decoding strategy which creates parallel search spaces for each monolingual and bilingual recognition tasks for the conventional CS ASR system. This strategy can be easily extended to E2E CTC ASR system to address the abovementioned data imbalance problem.…”

“…The parallel graphs used during decoding are characterized by the incorporated language model component, as they share the same token (T) and lexicon (L) components. This approach has been shown to outperform standard LM interpolation [26], that makes effective use of the text resources of the high-resourced language by creating three different search spaces with an identical acoustic model (AM). Monolingual and code-mixed utterances are decoded using best-matching subgraph, yielding improved monolingual recognition performance on the high-resourced language without any accuracy loss on the code-mixed utterances.…”

Code-Switch Speech Rescoring with Monolingual Data

“…The acoustic model is set up according to the paper [16], which contains both the Chinese syllable and the English phone as modeling units. We also apply the multi-graph decoding strategy [17], which contains code-switch and English 1 and the code-switch dataset as Table 2, and the English is trained by the transcript of librispeech. Besides, we implement the methods that optimizes the NGram language model in [7] and [18] as the baseline of the first-pass decoding, and these two methods together achieve 7% relative WER reduction.…”

The CSTR System for Multilingual and Code-Switching ASR Challenges for Low Resource Indian Languages

“…As an example, speakers of minority languages may use English terms when speaking in their native language. Therefore, there has been a lot of research trying to address this issue by focusing on acoustic modelling [10,11], language modelling [12,13], decoding [14] and data augmentation [15].…”