Learning words from sights and sounds: a computational model

Abstract

This paper presents an implemented computational model of word acquisition which learns directly from raw multimodal sensory input. Set in an information theoretic framework, the model acquires a lexicon by finding and statistically modeling consistent cross-modal structure. The model has been implemented in a system using novel speech processing, computer vision, and machine learning algorithms. In evaluations the model successfully performed speech segmentation, word discovery and visual categorization from spontaneous infant-directed speech paired with video images of single objects. These results demonstrate the possibility of using state-of-the-art techniques from sensory pattern recognition and machine learning to implement cognitive models which can process raw sensor data without the need for human transcription or labeling.

Ontology-Assisted Object Detection: Towards the Automatic Learning with Internet

“…In [16], an incremental version of Support Vector Machines is used to acquire visual categories. In the context of humanrobot interaction, some recent approaches also explore the combination of incremental learning and interaction with teachers to ground vocabulary about physical objects [10,5,13].…”

A Computational Model of Unsupervised Speech Segmentation for Correspondence Learning

“…A number of approaches to language acquisition in the phonetic realm frame the problem as recognizing entire words in longer utterances before phones are acquired (ten Bosch et al 2008;Roy and Pentland 2002;Werker and Curtin 2005). Presumably, phones would then be derived in a subsequent step from the recognized wordsalthough some authors raise the possibility that phones and segmentation into phones do not play any important role in language acquisition and that all learning happens on the level of words and utterances.…”

“…Blackburn and Young 1996;Roy and Pentland 2002;Coen 2006). However, these systems are not primarily concerned with segmentation and correspondence learning.…”

Intelligent Management of Data Driven Simulations to Support Model Building in the Social Sciences

“…Behaviour-based approaches are "data-driven" because the result of the learning process is determined largely by low-level features in the environment and less by any pre-defined knowledge in an ontology. Recent work in concept formation involving symbol grounding includes [12,13].…”