Multilingual representations for low resource speech recognition and keyword search

Cui, Jia; Kingsbury, Brian; Ramabhadran, Bhuvana; Sethy, Abhinav; Audhkhasi, Kartik; Cui, Xiaodong; Kislal, Ellen Eide; Mangu, Lidia; Nußbaum-Thom, Markus; Picheny, Michael; Tüske, Zoltán; Golik, Pavel; Schlüter, Ralf; Ney, Hermann; Gales, Mark J. F.; Knill, Kate; Ragni, Anton; Wang, Haipeng; Woodland, Philip C.

doi:10.1109/asru.2015.7404803

Cited by 79 publications

(48 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One way to re-use information extracted from other multilingual corpora is to use multilingual bottleneck features (BNFs), which has shown to perform well in conventional ASR as well as intrinsic evaluations [20,21,[27][28][29][30]. These features are typically obtained by training a deep neural network jointly on several languages for which labelled data is available.…”

Section: Bottleneck Featuresmentioning

confidence: 99%

ASR-Free CNN-DTW Keyword Spotting Using Multilingual Bottleneck Features for Almost Zero-Resource Languages

Menon

Kamper

Yılmaz

et al. 2018

6th Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU 2018)

View full text Add to dashboard Cite

We consider multilingual bottleneck features (BNFs) for nearly zero-resource keyword spotting. This forms part of a United Nations effort using keyword spotting to support humanitarian relief programmes in parts of Africa where languages are severely under-resourced. We use 1920 isolated keywords (40 types, 34 minutes) as exemplars for dynamic time warping (DTW) template matching, which is performed on a much larger body of untranscribed speech. These DTW costs are used as targets for a convolutional neural network (CNN) keyword spotter, giving a much faster system than direct DTW. Here we consider how available data from well-resourced languages can improve this CNN-DTW approach. We show that multilingual BNFs trained on ten languages improve the area under the ROC curve of a CNN-DTW system by 10.9% absolute relative to the MFCC baseline. By combining low-resource DTW-based supervision with information from well-resourced languages, CNN-DTW is a competitive option for low-resource keyword spotting.

show abstract

Section: Bottleneck Featuresmentioning

confidence: 99%

ASR-Free CNN-DTW Keyword Spotting Using Multilingual Bottleneck Features for Almost Zero-Resource Languages

Menon

Kamper

Yılmaz

et al. 2018

6th Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU 2018)

View full text Add to dashboard Cite

show abstract

“…Moreover, they significantly simplify infrastructure by supporting n languages with a single speech model rather than n individual models. Successful strategies for building multilingual acoustic models (AMs) include stacked bottleneck features [1][2][3][4], shared hidden layers [5,6], knowledge distillation [7], and multitask learning [8]. Building multilingual language models (LMs) has also been attempted recently (e.g.…”

Section: Introductionmentioning

confidence: 99%

Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model

et al. 2019

Self Cite

View full text Add to dashboard Cite

Multilingual end-to-end (E2E) models have shown great promise in expansion of automatic speech recognition (ASR) coverage of the world's languages. They have shown improvement over monolingual systems, and have simplified training and serving by eliminating language-specific acoustic, pronunciation, and language models. This work presents an E2E multilingual system which is equipped to operate in low-latency interactive applications, as well as handle a key challenge of real world data: the imbalance in training data across languages. Using nine Indic languages, we compare a variety of techniques, and find that a combination of conditioning on a language vector and training language-specific adapter layers produces the best model. The resulting E2E multilingual model achieves a lower word error rate (WER) than both monolingual E2E models (eight of nine languages) and monolingual conventional systems (all nine languages).

show abstract

“…There are two different ways to use the transcribed other source languages data. One way is to build hybrid multilingual deep neural network (MDNN), and then conduct the crosslingual transfer for target language [15], [17], [18], [21], [22]. Another way is to build bottleneck feature (BNF) extractor.…”

Section: Introductionmentioning

confidence: 99%

Submodular Based Unsupervised Data Selection

Zhang

2018

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYAutomatic speech recognition (ASR) and keyword search (KWS) have more and more found their way into our everyday lives, and their successes could boil down lots of factors. In these factors, large scale of speech data used for acoustic modeling is the key factor. However, it is difficult and time-consuming to acquire large scale of transcribed speech data for some languages, especially for low-resource languages. Thus, at low-resource condition, it becomes important with which transcribed data for acoustic modeling for improving the performance of ASR and KWS. In view of using acoustic data for acoustic modeling, there are two different ways. One is using the target language data, and another is using large scale of other source languages data for cross-lingual transfer. In this paper, we propose some approaches for efficient selecting acoustic data for acoustic modeling. For target language data, a submodular based unsupervised data selection approach is proposed. The submodular based unsupervised data selection could select more informative and representative utterances for manual transcription for acoustic modeling. For other source languages data, the high misclassified as target language based submodular multilingual data selection approach and knowledge based group multilingual data selection approach are proposed. When using selected multilingual data for multilingual deep neural network training for crosslingual transfer, it could improve the performance of ASR and KWS of target language. When comparing our proposed multilingual data selection approach with language identification based multilingual data selection approach, our proposed approach also obtains better effect. In this paper, we also analyze and compare the language factor and the acoustic factor influence on the performance of ASR and KWS. The influence of different scale of target language data on the performance of ASR and KWS at mono-lingual condition and cross-lingual condition are also compared and analyzed, and some significant conclusions can be concluded. key words: keyword spotting, submodular, multilingual data selection, language identification, recurrent neural network long short term memory

show abstract

Multilingual representations for low resource speech recognition and keyword search

Cited by 79 publications

References 38 publications

ASR-Free CNN-DTW Keyword Spotting Using Multilingual Bottleneck Features for Almost Zero-Resource Languages

ASR-Free CNN-DTW Keyword Spotting Using Multilingual Bottleneck Features for Almost Zero-Resource Languages

Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model

Submodular Based Unsupervised Data Selection

Contact Info

Product

Resources

About