Acoustic Word Embeddings for Zero-Resource Languages Using Self-Supervised Contrastive Learning and Multilingual Adaptation

Jacobs, Christiaan; Matusevych, Yevgen; Kamper, Herman

doi:10.1109/slt48900.2021.9383594

Cited by 18 publications

(18 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Word discrimination. (word-disc) is the task of detecting whether two speech segments correspond to the same or different words [33] and is commonly used to evaluate acoustic word embeddings and other acoustic representations [34][35][36][37]. We follow a typical evaluation protocol, where we label a pair of segments as "same word" if the cosine similarity between their word-level representations is above some threshold, and measure performance via the average precision as the threshold is varied.…”

Section: Analysis Methodsmentioning

confidence: 99%

Layer-wise Analysis of a Self-supervised Speech Representation Model

Pasad¹,

Chou²,

Livescu³

2021

Preprint

View full text Add to dashboard Cite

Recently proposed self-supervised learning approaches have been successful for pre-training speech representation models. The utility of these learned representations has been observed empirically, but not much has been studied about the type or extent of information encoded in the pre-trained representations themselves. Developing such insights can help understand the capabilities and limits of these models and enable the research community to more efficiently develop their usage for downstream applications. In this work, we begin to fill this gap by examining one recent and successful pre-trained model (wav2vec 2.0), via its intermediate representation vectors, using a suite of analysis tools. We use the metrics of canonical correlation, mutual information, and performance on simple downstream tasks with non-parametric probes, in order to (i) query for acoustic and linguistic information content, (ii) characterize the evolution of information across model layers, and (iii) understand how fine-tuning the model for automatic speech recognition (ASR) affects these observations. Our findings motivate modifying the fine-tuning protocol for ASR, which produces improved word error rates in a low-resource setting.

show abstract

Section: Analysis Methodsmentioning

confidence: 99%

Layer-wise Analysis of a Self-supervised Speech Representation Model

Pasad¹,

Chou²,

Livescu³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We use the CONTRASTIVERNN AWE model of [33]. 2 It performed the best of the model variants considered for multilingual transfer in [33].…”

Section: Acoustic Word Embedding Modelmentioning

confidence: 99%

“…We use the CONTRASTIVERNN AWE model of [33]. 2 It performed the best of the model variants considered for multilingual transfer in [33]. The model consists of an encoder recurrent neural network (RNN) that produces fixed-dimensional representations from variable-length speech segments.…”

Section: Acoustic Word Embedding Modelmentioning

confidence: 99%

“…The goal is to have the benefits of supervised learning by training a model on labelled data from multiple well-resourced languages, but to then apply the model to an unseen target zero-resource language without fine-tuning ita form of transductive transfer learning [32]. This multilingual transfer approach has been shown to outperform unsupervised monolingual AWE models [29,33].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multilingual Transfer of Acoustic Word Embeddings Improves When Training on Languages Related to the Target Zero-Resource Language

Jacobs¹,

Kamper²

2021

Interspeech 2021

Self Cite

View full text Add to dashboard Cite

Acoustic word embedding models map variable duration speech segments to fixed dimensional vectors, enabling efficient speech search and discovery. Previous work explored how embeddings can be obtained in zero-resource settings where no labelled data is available in the target language. The current best approach uses transfer learning: a single supervised multilingual model is trained using labelled data from multiple well-resourced languages and then applied to a target zero-resource language (without fine-tuning). However, it is still unclear how the specific choice of training languages affect downstream performance. Concretely, here we ask whether it is beneficial to use training languages related to the target. Using data from eleven languages spoken in Southern Africa, we experiment with adding data from different language families while controlling for the amount of data per language. In word discrimination and query-by-example search evaluations, we show that training on languages from the same family gives large improvements. Through finer-grained analysis, we show that training on even just a single related language gives the largest gain. We also find that adding data from unrelated languages generally doesn't hurt performance.

show abstract

“…Past works address this challenge with attempts to adapt the training vocabularies to new languages [78,79,80], convert the lexicons of other languages to the target language [81,51], or extend the vocabulary and fine-tune on the target language's data [82,83,84]. Another recent direction is pre-training the models to learn word-level acoustic representations [85,86,87]. Apart from our proposed ASR systems, any ASR system attempting to recognize languageindependent (i.e., phonetic) units, such as the multilingual allophone approach proposed in [13], would be suitable for performing phonetic inventory discovery.…”

Section: Related Workmentioning

confidence: 99%

Discovering Phonetic Inventories with Crosslingual Automatic Speech Recognition

Żelasko¹,

Feng²,

Moro-Velázquez³

et al. 2022

Preprint

View full text Add to dashboard Cite

The high cost of data acquisition makes Automatic Speech Recognition (ASR) model training problematic for most existing languages, including languages that do not even have a written script, or for which the phone inventories remain unknown. Past works explored multilingual training, transfer learning, as well as zero-shot learning in order to build ASR systems for these low-resource languages. While it has been shown that the pooling of resources from multiple languages is helpful, we have not yet seen a successful application of an ASR model to a language unseen during training. A crucial step in the adaptation of ASR from seen to unseen languages is the creation of the phone inventory of the unseen language.The ultimate goal of our work is to build the phone inventory of a language unseen during training in an unsupervised way without any knowledge about the language. In this paper, we 1) investigate the influence of different factors (i.e., model architecture, phonotactic model, type of speech representation) on phone recognition in an unknown language; 2) provide an analysis of which phones transfer well across languages and which do not in order to understand the limitations of and areas for further improvement for automatic phone inventory creation; and 3) present different methods to build a phone inventory of an unseen language in an unsupervised way. To that

show abstract

Acoustic Word Embeddings for Zero-Resource Languages Using Self-Supervised Contrastive Learning and Multilingual Adaptation

Cited by 18 publications

References 52 publications

Layer-wise Analysis of a Self-supervised Speech Representation Model

Layer-wise Analysis of a Self-supervised Speech Representation Model

Multilingual Transfer of Acoustic Word Embeddings Improves When Training on Languages Related to the Target Zero-Resource Language

Discovering Phonetic Inventories with Crosslingual Automatic Speech Recognition

Contact Info

Product

Resources

About