Multilingual and unsupervised subword modeling for zero-resource languages

Abstract

Unsupervised subword modeling aims to learn lowlevel representations of speech audio in "zero-resource" settings: that is, without using transcriptions or other resources from the target language (such as text corpora or pronunciation dictionaries). A good representation should capture phonetic content and abstract away from other types of variability, such as speaker differences and channel noise. Previous work in this area has primarily focused on learning from target language data only, and has been evaluated only intrinsically. Here we directly compare multiple methods, including some that use only target language speech data and some that use transcribed speech from other (non-target) languages, and we evaluate using two intrinsic measures as well as on a downstream unsupervised word segmentation and clustering task. We find that combining two existing target-language-only methods yields better features than either method alone. Nevertheless, even better results are obtained by extracting target language bottleneck features using a model trained on other languages. Cross-lingual training using just one other language is enough to provide this benefit, but multilingual training helps even more. In addition to these results, which hold across both intrinsic measures and the extrinsic task, we discuss the qualitative differences between the different types of learned features.

Multilingual Acoustic Word Embedding Models for Processing Zero-resource Languages

“…We perform our experiments on the GlobalPhone corpus of read speech [22]. As in [25], we treat six languages as our target zero-resource languages: Spanish (ES), Hausa (HA), Croatian (HR), Swedish (SV), Turkish (TR) and Mandarin (ZH). Each language has on average 16 hours of training, 2 hours of development and 2 hours of test data.…”

“…The area under this curve is used as final evaluation metric, referred to as the average precision (AP). We use the same specific evaluation setup as in [25].…”

“…This study is inspired by recent work showing the benefit of using multilingual bottleneck features as frame-level representations for zero-resource languages [23][24][25][26]. In [27], multilingual data were used in a similar way for discovering acoustic units.…”

Exploiting Cross-Lingual Speaker and Phonetic Diversity for Unsupervised Subword Modeling

“…Another approach seeks to use pre-trained outof-domain ASR systems to tokenize untranscribed in-domain speech and hence each frame is assigned with an ASR senone label [5], [14]. Fully unsupervised [13] or weakly supervised [15]- [17] methods for DNN training were also reported in the research on acoustic modeling for low-resource languages.…”

Analyzing ASR Pretraining for Low-Resource Speech-to-Text Translation

“…Although our main focus is monolingual pretraining, we also looked briefly at multilingual pretraining, inspired by recent work on multilingual ASR [29,30] and evidence that multilingual pretraining followed by fine-tuning on a distinct target language can improve ASR on the target language [11,31,32]. These experiments did not directly compare pretraining using a similar amount of monolingual data, but such a comparison was done by [33,34] in their work on learning feature representations for a target language with no transcribed data. They found a benefit for multilingual vs monolingual pretraining given the same amount of data.…”