Trilingual Semantic Embeddings of Visually Grounded Speech with Self-Attention Mechanisms

Ohishi, Yasunori; Kimura, Atsushi; Kawanishi, Takahito; Kashino, Kunio; Harwath, David; Glass, James

doi:10.1109/icassp40776.2020.9053428

Cited by 19 publications

(13 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Images and Spoken Captions. We fine-tune and evaluate our model on the Places Audio Caption dataset [4], which contains 100k images from the Places205 dataset [22] each with a spoken caption in Japanese [11] and Hindi [9]. We evaluate the performance on audio to image and image to audio retrieval using the standard recall metrics R@1, R@5, R@10.…”

Section: Methodsmentioning

confidence: 99%

“…We evaluate the performance on audio to image and image to audio retrieval using the standard recall metrics R@1, R@5, R@10. We follow the prior work [9,11] and report results on the validation sets of 1k images and spoken captions.…”

Section: Methodsmentioning

confidence: 99%

“…Harwath et al [9] collected 100k Hindi captions of Places images and proposed a bilingual audio-visual model. Building from this, Ohishi et al [11] collected 100k Japanese captions and proposed a trilingual model. Other work has proposed bilingual models with synthetic spoken captions [10] and image text taggers [32], clustering for bilingual image-audio dictionaries [33], and pair expansion methods for learning from multilingual captions of disjoint images [34].…”

Section: Related Workmentioning

confidence: 99%

“…We compare our approach to the state-of-the-art models for each dataset. While previous models are not trained on HowTo100M videos, some of them [9,11] are trained on images with parallel spoken captions in multiple languages. Our cascaded approach involves training on one language at a time, achieving large gains over prior baselines.…”

Section: Image Retrievalmentioning

confidence: 99%

See 3 more Smart Citations

Cascaded Multilingual Audio-Visual Learning from Videos

Rouditchenko¹,

Boggust²,

Harwath³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Image Retrievalmentioning

confidence: 99%

See 2 more Smart Citations

Cascaded Multilingual Audio-Visual Learning from Videos

Rouditchenko¹,

Boggust²,

Harwath³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Whether CNNbased approaches or RNN-based approaches are employed, all seem to segment individual words from the inputted spoken utterance (Harwath et al, 2016;. This result stands also for languages other than English, such as Hindi or Japanese (Harwath et al, 2018;Azuh et al, 2019;Ohishi et al, 2020). and , however, observed that not all layers encode wordlike units, suggesting that some layers specialise in lexical processing whereas some other do not encode such information.…”

Section: Introduction and Prior Workmentioning

confidence: 99%

Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speech

Havard¹,

Besacier²,

Chevrot³

2020

Proceedings of the 24th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

The language acquisition literature shows that children do not build their lexicon by segmenting the spoken input into phonemes and then building up words from them, but rather adopt a top-down approach and start by segmenting word-like units and then break them down into smaller units. This suggests that the ideal way of learning a language is by starting from full semantic units. In this paper, we investigate if this is also the case for a neural model of Visually Grounded Speech trained on a speech-image retrieval task. We evaluated how well such a network is able to learn a reliable speech-to-image mapping when provided with phone, syllable, or word boundary information. We present a simple way to introduce such information into an RNN-based model and investigate which type of boundary is the most efficient. We also explore at which level of the network's architecture such information should be introduced so as to maximise its performances. Finally, we show that using multiple boundary types at once in a hierarchical structure, by which low-level segments are used to recompose high-level segments, is beneficial and yields better results than using lowlevel or high-level segments in isolation.

show abstract