Improved acoustic word embeddings for zero-resource languages using multilingual transfer

Abstract

Acoustic word embeddings are fixed-dimensional representations of variable-length speech segments. Such embeddings can form the basis for speech search, indexing and discovery systems when conventional speech recognition is not possible. In zero-resource settings where unlabelled speech is the only available resource, we need a method that gives robust embeddings on an arbitrary language. Here we explore multilingual transfer: we train a single supervised embedding model on labelled data from multiple well-resourced languages and then apply it to unseen zero-resource languages. We consider three multilingual recurrent neural network (RNN) models: a classifier trained on the joint vocabularies of all training languages; a Siamese RNN trained to discriminate between same and different words from multiple languages; and a correspondence autoencoder (CAE) RNN trained to reconstruct word pairs. In a word discrimination task on six target languages, all of these models outperform state-of-the-art unsupervised models trained on the zero-resource languages themselves, giving relative improvements of more than 30% in average precision. When using only a few training languages, the multilingual CAE performs better, but with more training languages the other multilingual models perform similarly. Using more training languages is generally beneficial, but improvements are marginal on some languages. We present probing experiments which show that the CAE encodes more phonetic, word duration, language identity and speaker information than the other multilingual models.

Acoustic Word Embeddings for Zero-Resource Languages Using Self-Supervised Contrastive Learning and Multilingual Adaptation

“…Instead of relying on discovered words from the target zeroresource language, we can exploit labelled data from wellresourced languages to train a single multilingual supervised AWE model [31,32]. This model can then be applied to an unseen zero-resource language.…”

“…Experiments in [31] showed that multilingual versions of the CAE-RNN and SIAMESERNN outperform unsupervised monolingual variants. A multilingual CONTRASTIVERNN hasn't been considered in a previous study, as far as we know.…”

“…The idea is to train a supervised multilingual AWE model jointly on a number of well-resourced languages for which labelled data is available, but to then apply the model to an unseen zero-resource language. This multilingual transfer approach was found to outperform monolingual unsupervised learning approaches in [31,32].…”

“…A recent alternative for obtaining embeddings on a zeroresource language is to use multilingual transfer learning [29][30][31][32]. The idea is to train a supervised multilingual AWE model jointly on a number of well-resourced languages for which labelled data is available, but to then apply the model to an unseen zero-resource language.…”

A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings

“…To our knowledge, this is the first time an unsupervised acoustic word embedding approach outperforms DTW on the Buckeye corpus. The SiameseRNN is not competitive in this setting, potentially because it is more reliant on high-quality training pairs than the other models (which is supported by prior work [Kamper et al, 2020a]).…”