Investigating Bilingual Deep Neural Networks for Automatic Recognition of Code-switching Frisian Speech

Yılmaz, Emre; Heuvel, H. van den; Leeuwen, David A. van

doi:10.1016/j.procs.2016.04.044

Cited by 67 publications

(54 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this multilingual training scheme, the phones of each language are modeled separately, e.g., by appending a language identifier to every phone of a word based on the language of its lexicon. We refer the reader to [38] in which the impact of phone merging in the context of CS ASR is explored. The words in each lexicon are also tagged with language identifier to be able to evaluate CS detection accuracy.…”

Section: Multilingual Trainingmentioning

confidence: 99%

Code-switching detection using multilingual DNNS

Yılmaz

Heuvel

Leeuwen

2016

2016 IEEE Spoken Language Technology Workshop (SLT)

Self Cite

View full text Add to dashboard Cite

Automatic speech recognition (ASR) of code-switching speech requires careful handling of unexpected language switches that may occur in a single utterance. In this paper, we investigate the feasibility of using multilingually trained deep neural networks (DNN) for the ASR of Frisian speech containing code-switches to Dutch with the aim of building a robust recognizer that can handle this phenomenon. For this purpose, we train several multilingual DNN models on Frisian and two closely related languages, namely English and Dutch, to compare the impact of single-step and two-step multilingual DNN training on the recognition and code-switching detection performance. We apply bilingual DNN retraining on both target languages by varying the amount of training data belonging to the higher-resourced target language (Dutch). The recognition results show that the multilingual DNN training scheme with an initial multilingual training step followed by bilingual retraining provides recognition performance comparable to an oracle baseline recognizer that can employ language-specific acoustic models. We further show that we can detect code-switches at the word level with an equal error rate of around 17% excluding the deletions due to ASR errors.

show abstract

Section: Multilingual Trainingmentioning

confidence: 99%

Code-switching detection using multilingual DNNS

Yılmaz

Heuvel

Leeuwen

2016

2016 IEEE Spoken Language Technology Workshop (SLT)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Impact of CS and other kinds of language switches on the ASR systems has recently received research interest, resulting in multiple approaches for CS acoustic and language modeling [7][8][9][10][11][12][13][14][15][16]. The main focus of the FAME!…”

Section: Introductionmentioning

confidence: 99%

Multi-Graph Decoding for Code-Switching ASR

et al. 2019

Self Cite

View full text Add to dashboard Cite

In the FAME! Project, a code-switching (CS) automatic speech recognition (ASR) system for Frisian-Dutch speech is developed that can accurately transcribe the local broadcaster's bilingual archives with CS speech. This archive contains recordings with monolingual Frisian and Dutch speech segments as well as Frisian-Dutch CS speech, hence the recognition performance on monolingual segments is also vital for accurate transcriptions. In this work, we propose a multi-graph decoding and rescoring strategy using bilingual and monolingual graphs together with a unified acoustic model for CS ASR. The proposed decoding scheme gives the freedom to design and employ alternative search spaces for each (monolingual or bilingual) recognition task and enables the effective use of monolingual resources of the high-resourced mixed language in low-resourced CS scenarios. In our scenario, Dutch is the high-resourced and Frisian is the low-resourced language. We therefore use additional monolingual Dutch text resources to improve the Dutch language model (LM) and compare the performance of single-and multi-graph CS ASR systems on Dutch segments using larger Dutch LMs. The ASR results show that the proposed approach outperforms baseline single-graph CS ASR systems, providing better performance on the monolingual Dutch segments without any accuracy loss on monolingual Frisian and code-mixed segments.

show abstract

“…Further details of this corpus are given in [18]. In the last years, we have created two publicly available speech corpora for CS ASR [19] and speaker recognition [17] research using the manually annotated and raw data extracted from these archives. After having the complete archive digitized last year, we have created a third speech corpus that is designed for largescale SD research.…”

Section: Fame! Speaker Diarization Corpusmentioning

confidence: 99%

Large-Scale Speaker Diarization of Radio Broadcast Archives

Yılmaz¹,

Derinel²,

Zhou³

et al. 2019

Interspeech 2019

Self Cite

View full text Add to dashboard Cite

This paper describes our initial efforts to build a large-scale speaker diarization (SD) and identification system on a recently digitized radio broadcast archive from the Netherlands which has more than 6500 audio tapes with 3000 hours of Frisian-Dutch speech recorded between 1950-2016. The employed large-scale diarization scheme involves two stages: (1) tapelevel speaker diarization providing pseudo-speaker identities and (2) speaker linking to relate pseudo-speakers appearing in multiple tapes. Having access to the speaker models of several frequently appearing speakers from the previously collected FAME! speech corpus, we further perform speaker identification by linking these known speakers to the pseudo-speakers identified at the first stage. In this work, we present a recently created longitudinal and multilingual SD corpus designed for large-scale SD research and evaluate the performance of a new speaker linking system using x-vectors with PLDA to quantify cross-tape speaker similarity on this corpus. The performance of this speaker linking system is evaluated on a small subset of the archive which is manually annotated with speaker information. The speaker linking performance reported on this subset (53 hours) and the whole archive (3000 hours) is compared to quantify the impact of scaling up in the amount of speech data.

show abstract

Investigating Bilingual Deep Neural Networks for Automatic Recognition of Code-switching Frisian Speech

Cited by 67 publications

References 23 publications

Code-switching detection using multilingual DNNS

Code-switching detection using multilingual DNNS

Multi-Graph Decoding for Code-Switching ASR

Large-Scale Speaker Diarization of Radio Broadcast Archives

Contact Info

Product

Resources

About