Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech

Abstract

In this paper, we study how word-like units are represented and activated in a recurrent neural model of visually grounded speech. The model used in our experiments is trained to project an image and its spoken description in a common representation space. We show that a recurrent model trained on spoken sentences implicitly segments its input into word-like units and reliably maps them to their correct visual referents. We introduce a methodology originating from linguistics to analyse the representation learned by neural networks -the gating paradigm -and show that the correct representation of a word is only activated if the network has access to first phoneme of the target word, suggesting that the network does not rely on a global acoustic pattern. Furthermore, we find out that not all speech frames (MFCC vectors in our case) play an equal role in the final encoded representation of a given word, but that some frames have a crucial effect on it. Finally, we suggest that word representation could be activated through a process of lexical competition. Related WorkIn this section we explore what is known about word recognition in humans. We then review recent works related to the representation of lan-arXiv:1909.08491v1 [cs.CL]

Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? - A computational investigation

“…In terms of our more detailed linguistic analyses, the present findings largely align with the earlier literature on investigating linguistic units in VGS models (e.g., Chrupała et al, 2017;Alishahi et al, 2017;Havard et al, 2019aHavard et al, , 2019bMerkx et al, 2019). However, the present study is the first one to show that broadly similar learning takes place in different model architectures (convolutional and recurrent)…”

“…This is in line with the knowledge that infant early vocabulary tends to predominantly consist of concrete nouns. In another study, Havard et al (2019b) examined the influence of different input data characteristics in a word recognition task by feeding the VGS model with synthesized isolated words with varying characteristics.…”

“…This is in line with the knowledge that infant early vocabulary tends to predominantly consist of concrete nouns. In another study, Havard et al (2019b) examined the influence of different input data characteristics in a word recognition task by feeding the VGS model with synthesized isolated words with varying characteristics. They observed a moderate correlation between word recognition accuracy and frequency of the words in training data, and a weak correlation for image-related factors such as visual object size and saliency.…”

Grounding ‘Grounding’ in NLP

“…• Non-textual modality: Projection to a joint semantic space is used in spoken image captioning Havard et al, 2019), bicoding for learning image attributes (Silberer and Lapata, 2014), representation learning of images (Zarrieß and Schlangen, 2017) and speech (Vijayakumar et al, 2017).…”

Text-Free Image-to-Speech Synthesis Using Learned Segmental Units

“…A rich body of work has recently emerged investigating representation learning for speech using visual grounding objectives (Synnaeve et al, 2014;Harwath and Glass, 2015;Harwath et al, 2016;Kamper et al, 2017;Havard et al, 2019a;Merkx et al, 2019;Scharenborg et al, 2018;Hsu and Glass, 2018a;Kamper et al, 2018;Ilharco et al, 2019;Eloff et al, 2019), as well as how word-like and subword-like linguistic units can be made to emerge within these models (Harwath and Glass, 2017;Drexler and Glass, 2017;Havard et al, 2019b;Harwath et al, 2020). So far, these efforts have predominantly focused on inference, where the goal is to learn a mapping from speech waveforms to a semantic embedding space.…”