Semantic Query-by-example Speech Search Using Visual Grounding

Kamper, Herman; Anastassiou, Aristotelis; Livescu, Karen

doi:10.1109/icassp.2019.8683275

Cited by 19 publications

(11 citation statements)

References 36 publications

(65 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of approaches have been put forward for AWE-based QbE [1,40,41]. Here we use the simplified approach from [42]. Using an AWE model, we first embed the query segment.…”

Section: Query-by-example Speech Searchmentioning

confidence: 99%

Multilingual Transfer of Acoustic Word Embeddings Improves When Training on Languages Related to the Target Zero-Resource Language

Jacobs¹,

Kamper²

2021

Interspeech 2021

Self Cite

View full text Add to dashboard Cite

Acoustic word embedding models map variable duration speech segments to fixed dimensional vectors, enabling efficient speech search and discovery. Previous work explored how embeddings can be obtained in zero-resource settings where no labelled data is available in the target language. The current best approach uses transfer learning: a single supervised multilingual model is trained using labelled data from multiple well-resourced languages and then applied to a target zero-resource language (without fine-tuning). However, it is still unclear how the specific choice of training languages affect downstream performance. Concretely, here we ask whether it is beneficial to use training languages related to the target. Using data from eleven languages spoken in Southern Africa, we experiment with adding data from different language families while controlling for the amount of data per language. In word discrimination and query-by-example search evaluations, we show that training on languages from the same family gives large improvements. Through finer-grained analysis, we show that training on even just a single related language gives the largest gain. We also find that adding data from unrelated languages generally doesn't hurt performance.

show abstract

“…A number of approaches have been put forward for AWE-based QbE [1,40,41]. Here we use the simplified approach from [42]. Using an AWE model, we first embed the query segment.…”

Section: Query-by-example Speech Searchmentioning

confidence: 99%

Multilingual Transfer of Acoustic Word Embeddings Improves When Training on Languages Related to the Target Zero-Resource Language

Jacobs¹,

Kamper²

2021

Interspeech 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…This makes the method effective as a semantic keyword spotter. This work has led to a number of follow-ups and extensions (e.g., Kamper & Roth, 2018;Kamper, Anastassiou, & Livescu, 2019;Kamper, Shakhnarovich, & Livescu, 2019;Olaleye, van Niekerk, & Kamper, 2020, ).…”

Section: Abstractpottingmentioning

confidence: 99%

Visually Grounded Models of Spoken Language: A Survey of Datasets, Architectures and Evaluation Techniques

Chrupała¹

2022

jair

View full text Add to dashboard Cite

This survey provides an overview of the evolution of visually grounded models of spoken language over the last 20 years. Such models are inspired by the observation that when children pick up a language, they rely on a wide range of indirect and noisy clues, crucially including signals from the visual modality co-occurring with spoken utterances. Several fields have made important contributions to this approach to modeling or mimicking the process of learning language: Machine Learning, Natural Language and Speech Processing, Computer Vision and Cognitive Science. The current paper brings together these contributions in order to provide a useful introduction and overview for practitioners in all these areas. We discuss the central research questions addressed, the timeline of developments, and the datasets which enabled much of this work. We then summarize the main modeling architectures and offer an exhaustive overview of the evaluation metrics and analysis techniques.

show abstract

“…This tagger has an output of 1000 image classes, but here we use a system vocabulary corresponding to W = 67 unique keyword types. This set of keywords is the same set used in [27,39], and includes words such as 'children', 'young', 'swimming', 'snowy' and 'riding'. The procedure used to select these keywords are detailed in [27]; it includes a human reviewer agreement step, which reduced the original set from 70 to 67 words.…”

Section: Datamentioning

confidence: 99%

Attention-Based Keyword Localisation in Speech Using Visual Grounding

Olaleye¹,

Kamper²

2021

Interspeech 2021

Self Cite

View full text Add to dashboard Cite

Visually grounded speech models learn from images paired with spoken captions. By tagging images with soft text labels using a trained visual classifier with a fixed vocabulary, previous work has shown that it is possible to train a model that can detect whether a particular text keyword occurs in speech utterances or not. Here we investigate whether visually grounded speech models can also do keyword localisation: predicting where, within an utterance, a given textual keyword occurs without any explicit text-based or alignment supervision. We specifically consider whether incorporating attention into a convolutional model is beneficial for localisation. Although absolute localisation performance with visually supervised models is still modest (compared to using unordered bag-of-word text labels for supervision), we show that attention provides a large gain in performance over previous visually grounded models. As in many other speech-image studies, we find that many of the incorrect localisations are due to semantic confusions, e.g. locating the word 'backstroke' for the query keyword 'swimming'.

show abstract

Semantic Query-by-example Speech Search Using Visual Grounding

Cited by 19 publications

References 36 publications

Multilingual Transfer of Acoustic Word Embeddings Improves When Training on Languages Related to the Target Zero-Resource Language

Multilingual Transfer of Acoustic Word Embeddings Improves When Training on Languages Related to the Target Zero-Resource Language

Visually Grounded Models of Spoken Language: A Survey of Datasets, Architectures and Evaluation Techniques

Attention-Based Keyword Localisation in Speech Using Visual Grounding

Contact Info

Product

Resources

About