Code-Switching Detection with Data-Augmented Acoustic and Language Models

Yılmaz, Emre; Heuvel, H. van den; Leeuwen, David A. van

doi:10.21437/sltu.2018-27

Cited by 8 publications

(7 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Firstly, in terms of data augmentation, we boost the size of CS corpus by synthetically generating CS text using a well-trained long short-term memory (LSTM) language model. Similar techniques are also proposed in [27,28]. However, in [27] a sentence level aligned parallel corpus is available, thus synthetic CS data can be generated based on word or phrase alignment between the parallel sentences and guided by linguistic rules.…”

Section: Cs Rnn Language Modelingmentioning

confidence: 99%

End-to-End Code-Switching ASR for Low-Resourced Language Pairs

Yue

Lee

Yılmaz

et al. 2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

Self Cite

View full text Add to dashboard Cite

Despite the significant progress in end-to-end (E2E) automatic speech recognition (ASR), E2E ASR for low resourced code-switching (CS) speech has not been well studied. In this work, we describe an E2E ASR pipeline for the recognition of CS speech in which a low-resourced language is mixed with a high resourced language. Low-resourcedness in acoustic data hinders the performance of E2E ASR systems more severely than the conventional ASR systems. To mitigate this problem in the transcription of archives with code-switching Frisian-Dutch speech, we integrate a designated decoding scheme and perform rescoring with neural network-based language models to enable better utilization of the available textual resources. We first incorporate a multi-graph decoding approach which creates parallel search spaces for each monolingual and mixed recognition tasks to maximize the utilization of the textual resources from each language. Further, language model rescoring is performed using a recurrent neural network pre-trained with cross-lingual embedding and further adapted with the limited amount of in-domain CS text. The ASR experiments demonstrate the effectiveness of the described techniques in improving the recognition performance of an E2E CS ASR system in a low-resourced scenario.

show abstract

Section: Cs Rnn Language Modelingmentioning

confidence: 99%

End-to-End Code-Switching ASR for Low-Resourced Language Pairs

Yue

Lee

Yılmaz

et al. 2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Moreover, we have created CS text, which is almost nonexistent, in multiple ways providing perplexity reductions on the development and test set transcriptions. The data-augmented models have been shown to provide better CS detection in terms of equal error rate, but have the tendency to hypothesize much more language switches compared to the human annotations [21].…”

Section: Baseline Approach: Time Alignment Of Cs Asr Outputmentioning

confidence: 99%

“…In our latest work, we have investigated this code-switching detection performance using data-augmented CS ASR systems and observed that this technique suffers from over-switching due to numerous very short spurious language switches [21]. To cope with shortcoming, this paper introduces a new method for code-switching detection which uses frame-level language posteriors which are created using a CS ASR system.…”

Section: Introductionmentioning

confidence: 99%

Code-Switching Detection Using ASR-Generated Language Posteriors

Wang

Yılmaz

Derinel³

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

Code-switching (CS) detection refers to the automatic detection of language switches in code-mixed utterances. This task can be achieved by using a CS automatic speech recognition (ASR) system that can handle such language switches. In our previous work, we have investigated the code-switching detection performance of the Frisian-Dutch CS ASR system by using the time alignment of the most likely hypothesis and found that this technique suffers from over-switching due to numerous very short spurious language switches. In this paper, we propose a novel method for CS detection aiming to remedy this shortcoming by using the language posteriors which are the sum of the framelevel posteriors of phones belonging to the same language. The CS ASR-generated language posteriors contain more complete language-specific information on frame level compared to the time alignment of the ASR output. Hence, it is expected to yield more accurate and robust CS detection. The CS detection experiments demonstrate that the proposed language posterior-based approach provides higher detection accuracy than the baseline system in terms of equal error rate. Moreover, a detailed CS detection error analysis reveals that using language posteriors reduces the false alarms and results in more robust CS detection.

show abstract

“…To alleviate the problem of code-switching data shortage, monolingual data and data augment technology are usually used [10,11,12,13,14]. For the monolingual, some E2E models have achieved great performance with large-scale data.…”

Section: Introductionmentioning

confidence: 99%

Decoupling Pronunciation and Language for End-to-End Code-Switching Automatic Speech Recognition

Zhang

Tian

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Despite the recent significant advances witnessed in end-to-end (E2E) ASR system for code-switching, hunger for audio-text paired data limits the further improvement of the models' performance. In this paper, we propose a decoupled transformer model to use monolingual paired data and unpaired text data to alleviate the problem of code-switching data shortage. The model is decoupled into two parts: audio-to-phoneme (A2P) network and phoneme-to-text (P2T) network. The A2P network can learn acoustic pattern scenarios using large-scale monolingual paired data. Meanwhile, it generates multiple phoneme sequence candidates for single audio data in real time during the training process. Then the generated phoneme-text paired data is used to train the P2T network. This network can be pre-trained with large amounts of external unpaired text data. By using monolingual data and unpaired text data, the decoupled transformer model reduces the high dependency on code-switching paired training data of E2E model to a certain extent. Finally, the two networks are optimized jointly through attention fusion. We evaluate the proposed method on the public Mandarin-English codeswitching dataset. Compared with our transformer baseline, the proposed method achieves 18.14% relative mix error rate reduction.

show abstract

Code-Switching Detection with Data-Augmented Acoustic and Language Models

Cited by 8 publications

References 24 publications

End-to-End Code-Switching ASR for Low-Resourced Language Pairs

End-to-End Code-Switching ASR for Low-Resourced Language Pairs

Code-Switching Detection Using ASR-Generated Language Posteriors

Decoupling Pronunciation and Language for End-to-End Code-Switching Automatic Speech Recognition

Contact Info

Product

Resources

About