Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings

Settle, Shane; Levin, Keith; Kamper, Herman; Livescu, Karen

doi:10.21437/interspeech.2017-1592

Cited by 67 publications

(71 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, current methods require large amounts of transcribed data, which are available only for a small fraction of the world's languages [1]. This has prompted work on speech models that, instead of using exact transcriptions, can learn from weaker forms of supervision, e.g., known word pairs [2], [3], translation text [4]- [6], or unordered word labels [7]. The motivation for much of this work is that, even when high-quality ASR is infeasible, it may still be possible to learn low-resource models for practical tasks like retrieval and keyword prediction.…”

Section: Introductionmentioning

confidence: 99%

Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech

Kamper

Shakhnarovich

Livescu

2019

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

There is growing interest in models that can learn from unlabelled speech paired with visual context. This setting is relevant for low-resource speech processing, robotics, and human language acquisition research. Here we study how a visually grounded speech model, trained on images of scenes paired with spoken captions, captures aspects of semantics. We use an external image tagger to generate soft text labels from images, which serve as targets for a neural model that maps untranscribed speech to (semantic) keyword labels. We introduce a newly collected data set of human semantic relevance judgements and an associated task, semantic speech retrieval, where the goal is to search for spoken utterances that are semantically relevant to a given text query. Without seeing any text, the model trained on parallel speech and images achieves a precision of almost 60% on its top ten semantic retrievals. Compared to a supervised model trained on transcriptions, our model matches human judgements better by some measures, especially in retrieving non-verbatim semantic matches. We perform an extensive analysis of the model and its resulting representations. 1

show abstract

Section: Introductionmentioning

confidence: 99%

Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech

Kamper

Shakhnarovich

Livescu

2019

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our approach to semantic QbE is embedding-based: We learn an embedding function that maps from segments of speech-queries, search utterances, or sub-segments of search utterances-to fixed-dimensional vectors; we search for semantic matches by finding the minimum distance between query and search utterance embedding vectors. In this respect our approach is similar to those in recent embedding-based QbE work [14][15][16][17], and also some embedding-based spoken term detection work [18]. The key difference is that our embedding function must be learned in such a way that similar embedding vectors are semantically rather than phonetically similar.…”

Section: Introductionmentioning

confidence: 87%

“…The audio comprises around 37 hours of active speech in total. 67 keyword 1 While [14,16] use an approximate nearest neighbour search procedure, we use exhaustive search here. types are selected randomly from transcriptions of the training portion of the corpus.…”

Section: Experimental Setup and Evaluationmentioning

confidence: 99%

Semantic Query-by-example Speech Search Using Visual Grounding

Kamper

Anastassiou

Livescu

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

A number of recent studies have started to investigate how speech systems can be trained on untranscribed speech by leveraging accompanying images at training time. Examples of tasks include keyword prediction and within-and acrossmode retrieval. Here we consider how such models can be used for query-by-example (QbE) search, the task of retrieving utterances relevant to a given spoken query. We are particularly interested in semantic QbE, where the task is not only to retrieve utterances containing exact instances of the query, but also utterances whose meaning is relevant to the query. We follow a segmental QbE approach where variable-duration speech segments (queries, search utterances) are mapped to fixeddimensional embedding vectors. We show that a QbE system using an embedding function trained on visually grounded speech data outperforms a purely acoustic QbE system in terms of both exact and semantic retrieval performance.

show abstract

“…From a scientific perspective, discrete representations could be useful in cognitive models to study phonetic category learning in human infants [15][16][17][18]. From a technology perspective, such features could be used in downstream speech applications requiring symbolic or sparse input, e.g., for faster retrieval in speech search systems [19,20]. Here we consider the downstream task of speech synthesis within the context of the ZRSC'19.…”

Section: Introductionmentioning

confidence: 99%

Unsupervised Acoustic Unit Discovery for Speech Synthesis Using Discrete Latent-Variable Neural Networks

et al. 2019

Self Cite

View full text Add to dashboard Cite

For our submission to the ZeroSpeech 2019 challenge, we apply discrete latent-variable neural networks to unlabelled speech and use the discovered units for speech synthesis. Unsupervised discrete subword modelling could be useful for studies of phonetic category learning in infants or in low-resource speech technology requiring symbolic input. We use an autoencoder (AE) architecture with intermediate discretisation. We decouple acoustic unit discovery from speaker modelling by conditioning the AE's decoder on the training speaker identity. At test time, unit discovery is performed on speech from an unseen speaker, followed by unit decoding conditioned on a known target speaker to obtain reconstructed filterbanks. This output is fed to a neural vocoder to synthesise speech in the target speaker's voice. For discretisation, categorical variational autoencoders (CatVAEs), vectorquantised VAEs (VQ-VAEs) and straight-through estimation are compared at different compression levels on two languages. Our final model uses convolutional encoding, VQ-VAE discretisation, deconvolutional decoding and an FFTNet vocoder. We show that decoupled speaker conditioning intrinsically improves discrete acoustic representations, yielding competitive synthesis quality compared to the challenge baseline.

show abstract

Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings

Cited by 67 publications

References 24 publications

Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech

Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech

Semantic Query-by-example Speech Search Using Visual Grounding

Unsupervised Acoustic Unit Discovery for Speech Synthesis Using Discrete Latent-Variable Neural Networks

Contact Info

Product

Resources

About