Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? - A computational investigation

Abstract

Decades of research has studied how language learning infants learn to discriminate speech sounds, segment words, and associate words with their meanings. While gradual development of such capabilities is unquestionable, the exact nature of these skills and the underlying mental representations yet remains unclear. In parallel, computational studies have shown that basic comprehension of speech can be achieved by statistical learning between speech and concurrent referentially ambiguous visual input. These models can operate without prior linguistic knowledge such as representations of linguistic units, and without learning mechanisms specifically targeted at such units. This has raised the question whether knowledge of linguistic units, such as phone(me)s, syllables, and words, could actually emerge as latent representations supporting the translation between speech and representations in other modalities, and instead of the units ever being proximal learning goals for the learner. In this study, formulate this idea as the so-called latent language hypothesis (LLH), connecting linguistic representation learning to general predictive processing within and across sensory modalities. We review the extent that the audiovisual aspect of LLH is supported by the existing computational studies. We then explore LLH further in extensive learning simulations with different neural network models for audiovisual cross-situational learning, and comparing learning from both synthetic and real speech data. We investigate whether the latent representations learned by the networks reflect phonetic, syllabic, or lexical structure of input speech by utilizing an array of complementary evaluation metrics related to linguistic selectivity and temporal characteristics of the representations. As a result, we find that representations associated with phonetic, syllabic, and lexical units of speech indeed emerge from the audiovisual learning process. The finding is also robust against variations in model architecture or characteristics of model training and testing data. The results suggest that cross-modal and cross-situational learning may, in principle, assist in early language development much beyond just enabling association of acoustic word forms to their referential meanings.

Visually Grounded Models of Spoken Language: A Survey of Datasets, Architectures and Evaluation Techniques

“…This suggests that utterances are implicitly segmented into phonemes within this architecture. These findings were partially corroborated by Khorrami and Räsänen (2021), with the proviso of rather lower scores and the fact that implicit phoneme segmentation is also present to a large extent in activations from untrained models, and thus is not fully due to learning, but simply to network dynamics.…”

“…An alternative approach which is easier to apply to human speech was proposed by Khorrami and Räsänen (2021) and involves using automatically computed pairwise sentence similarities derived from a text-based model as the proxy for human similarity judgments of semantic relatedness. The automatic semantic relatedness score (SRS) is based on word-word similarity scores as given by Word2Vec (Mikolov, Chen, Corrado, & Dean, 2013) embedding vectors and defined as follows:…”

Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics

“…Our goal is to go beyond these analysis to test specific semantic phenomena as we did here with the Abstract Scenes dataset. Another step towards more naturalistic input is the use speech input instead of text Khorrami and Räsänen, 2021).…”

“…As commonly applied in other multimodal XSL work Khorrami and Räsänen, 2021), we assume that the visual system of the learner has already been developed to some degree and thus use a CNN pre-trained on ImageNet (Russakovsky et al, 2015) (but discard the final classification layer) to encode the images. Specifically, we use a ResNet 50 2 (He et al, 2016) to encode the images and train a linear embedding layer that maps the output of the pre-final layer of the CNN into the joint embedding space.…”

Modeling the Interaction Between Perception-Based and Production-Based Learning in Children's Early Acquisition of Semantic Knowledge

“…As commonly applied in other multimodal XSL work(Chrupała et al, 2015;Khorrami and Räsänen, 2021).6 WhileVinyals et al (2015) fed the image features only at the first timestep into the LSTM, here we feed it at every timestep as this showed to improve performance on our evaluation substantially. An explanation could be that when feeding the image features only at the first timestep the model gradually forgets about the input, and relies more on the language modeling task of next-word prediction, which does not aid the learning of visually-grounded semantics.…”