Improved Acoustic Word Embeddings for Zero-Resource Languages Using Multilingual Transfer

Abstract

Acoustic word embeddings are fixed-dimensional representations of variable-length speech segments. Such embeddings can form the basis for speech search, indexing and discovery systems when conventional speech recognition is not possible. In zero-resource settings where unlabelled speech is the only available resource, we need a method that gives robust embeddings on an arbitrary language. Here we explore multilingual transfer: we train a single supervised embedding model on labelled data from multiple well-resourced languages and then apply it to unseen zero-resource languages. We consider three multilingual recurrent neural network (RNN) models: a classifier trained on the joint vocabularies of all training languages; a Siamese RNN trained to discriminate between same and different words from multiple languages; and a correspondence autoencoder (CAE) RNN trained to reconstruct word pairs. In a word discrimination task on six target languages, all of these models outperform state-of-the-art unsupervised models trained on the zero-resource languages themselves, giving relative improvements of more than 30% in average precision. When using only a few training languages, the multilingual CAE performs better, but with more training languages the other multilingual models perform similarly. Using more training languages is generally beneficial, but improvements are marginal on some languages. We present probing experiments which show that the CAE encodes more phonetic, word duration, language identity and speaker information than the other multilingual models.

Multilingual Transfer of Acoustic Word Embeddings Improves When Training on Languages Related to the Target Zero-Resource Language

“…In the zero-resource setting we don't have labelled data in the target language to construct the positive and negative word pairs required for training. We therefore follow the approach of [29], and train a multilingual model on ground truth word pairs (extracted from forced alignments) from a number of languages for which we have labelled data. Subsequently, at test time, we apply the encoder RNN from the multilingual model to extract AWEs for speech from the target zero-resource language.…”

“…However, there still exists a large performance gap between these unsupervised models and their supervised counterparts [11,26]. A recent alternative for obtaining AWEs on a zero-resource language is to use multilingual transfer learning [27][28][29][30][31]. The goal is to have the benefits of supervised learning by training a model on labelled data from multiple well-resourced languages, but to then apply the model to an unseen target zero-resource language without fine-tuning ita form of transductive transfer learning [32].…”

“…The goal is to have the benefits of supervised learning by training a model on labelled data from multiple well-resourced languages, but to then apply the model to an unseen target zero-resource language without fine-tuning ita form of transductive transfer learning [32]. This multilingual transfer approach has been shown to outperform unsupervised monolingual AWE models [29,33].…”

“…Although there is clear benefit in applying multilingual AWE models to an unseen zero-resource language, it is still unclear how the particular choice of training languages affects subsequent performance. Preliminary experiments [29] show improved scores when training a monolingual model on one language and applying it to another from the same family. But this has not been investigated systematically and there are still several unanswered questions: Does the benefit of training on related languages diminish as we train on more languages (which might or might not come from the same family as the target zero-resource language)?…”

Detection of Consonant Errors in Disordered Speech Based on Consonant-vowel Segment Embedding

“…The comparison between a pair of segments is carried out based on speech embedding, which is a fixed-dimension representation that encodes acoustic information of speech segments into a low-dimension space. It allows flexible modeling and processing of speech segments of variable length for different downstream tasks, e.g., spoken term detection and discovery [14], pathological speech classification [15], prediction of speech intelligibility score [16], etc. A similarity or distance score can be calculated on each pair of embeddings.…”