Deep convolutional acoustic word embeddings using word-pair side information

Abstract

Recent studies have been revisiting whole words as the basic modelling unit in speech recognition and query applications, instead of phonetic units. Such whole-word segmental systems rely on a function that maps a variable-length speech segment to a vector in a fixed-dimensional space; the resulting acoustic word embeddings need to allow for accurate discrimination between different word types, directly in the embedding space. We compare several old and new approaches in a word discrimination task. Our best approach uses side information in the form of known word pairs to train a Siamese convolutional neural network (CNN): a pair of tied networks that take two speech segments as input and produce their embeddings, trained with a hinge loss that separates same-word pairs and different-word pairs by some margin. A word classifier CNN performs similarly, but requires much stronger supervision. Both types of CNNs yield large improvements over the best previously published results on the word discrimination task.Index Terms-Acoustic word embeddings, segmental acoustic models, fixed-dimensional representations, query-by-example search.

Multilingual Acoustic Word Embedding Models for Processing Zero-resource Languages

“…Several supervised and unsupervised acoustic embedding methods have been proposed. Supervised methods include convolutional [11][12][13] and recurrent neural network (RNN) models [14][15][16][17], trained with discriminative classification and contrastive losses. Unsupervised methods include using distances to a fixed reference set [10] and unsupervised autoencoding RNNs [18][19][20].…”

Additional Shared Decoder on Siamese Multi-View Encoders for Learning Acoustic Word Embeddings

“…Before the experiment, we implemented the baseline multiview approach [13] and trained it with the model and dataset provided by the authors to verify the performance improvement compared to the single-view approaches [9,10]. Then we established our initial model parameters as the same with the retuned baseline model on the WSJ dataset.…”

“…The first task is acoustic word discrimination, where we are given two word segments to determine whether they match or not. This task is equivalent to the objective of the single-view approach and has been used in prior papers [9,10,11,12,14,17]. We regard this task as our main evaluation task for training the proposed and baseline network architectures.…”

Exploring the use of acoustic embeddings in neural machine translation

“…Moreover, we will also investigate the use of different types of acoustic embeddings, such as those derived from siamese networks [24], that try to preserve distance of words both semantically and in acoustic space.…”