Acoustic and Textual Data Augmentation for Improved ASR of Code-Switching Speech

Abstract

In this paper, we describe several techniques for improving the acoustic and language model of an automatic speech recognition (ASR) system operating on code-switching (CS) speech. We focus on the recognition of Frisian-Dutch radio broadcasts where one of the mixed languages, namely Frisian, is an underresourced language. In previous work, we have proposed several automatic transcription strategies for CS speech to increase the amount of available training speech data. In this work, we explore how the acoustic modeling (AM) can benefit from monolingual speech data belonging to the high-resourced mixed language. For this purpose, we train state-of-the-art AMs, which were ineffective due to lack of training data, on a significantly increased amount of CS speech and monolingual Dutch speech. Moreover, we improve the language model (LM) by creating code-switching text, which is in practice almost nonexistent, by (1) generating text using recurrent LMs trained on the transcriptions of the training CS speech data, (2) adding the transcriptions of the automatically transcribed CS speech data and (3) translating Dutch text extracted from the transcriptions of a large Dutch speech corpora. We report significantly improved CS ASR performance due to the increase in the acoustic and textual training data.

Code-Switching Detection Using ASR-Generated Language Posteriors

“…CS ASR employs a bilingual acoustic model that captures the phonetic characteristics of both languages and a bilingual language model (LM) which can assign probabilities to code-mixed word sequences as well as monolingual word sequences from both languages. The current system uses data-augmented models described in [24]. The acoustic model is trained on automatically transcribed data from the same archive and a large amount of monolingual data from the high-resourced language (Dutch) together with the manually transcribed data form the FAME!…”

“…training data is the only source of CS text and contains 140k words. The remaining CS text is automatically generated as described in [24].…”

“…The LM used for the baseline CS detection system is a standard bilingual 3-gram with interpolated Kneser-Ney smoothing. Further details are provided in [24]. We compute phone posteriors from the denominator graph (created using a phone LM estimated from the phone alignments of the training data) of the chain model and map them to phones using the existing implementation in Kaldi (nnet3-chain-compute-post).…”

Large-Scale Speaker Diarization of Radio Broadcast Archives

“…Project, we have developed a spoken document retrieval system for the radio broadcast archives of Omrop Fryslân (Frisian Broadcast), the regional public broadcaster of the province Fryslân in the Netherlands. This system relies on automatically generated transcriptions hypothesized by a code-switching automatic speech recognition system [16] and speaker labels generated by a modern speaker recognition system developed using the resources [17] with the ultimate goal of making these archives searchable.…”

Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition

“…The upper panel summarizes the number of words from each language subset. The middle panel provides the results of state-of-the-art ANN achitecturesYılmaz et al, 2018) for reference purposes and the lower panel presents the results achieved by the ANN and SNN models in this work (AM: acoustic model).…”