Learning semantic components from subsymbolic multimodal perception

Mangin, Olivier; Oudeyer, Pierre-Yves

doi:10.1109/devlrn.2013.6652563

Cited by 19 publications

(16 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many researchers have investigated methods for robots to learn relationships among multimodal sensory data [12]- [16]. The purpose of these studies was to predict one modality from another by using models.…”

Section: Related Workmentioning

confidence: 99%

Online Algorithm for Robots to Learn Object Concepts and Language Model

Nishihara

Nagaoka

Nagai

2017

IEEE Trans. Cogn. Dev. Syst.

View full text Add to dashboard Cite

Humans form concept of objects by classifying them into categories, and acquire language by simultaneously interacting with others. Thus, the meaning of a word can be learned by connecting a recognized word to its corresponding concept. We consider this ability important for robots to flexibly develop knowledge of language and concepts. In this paper, we propose an online algorithm for robots to acquire knowledge of natural language and learn object concepts. A robot learns the language model from word sequences, which are obtained by the segmentation of phoneme sequences provided by a user, by using unsupervised word segmentation each time it is provided with a new object. Moreover, the robot acquires object concepts using these word sequences as well as multimodal information obtained by observing objects. The crucial aspect of our model is the interdependence of words and concepts: there is a high probability that the same words will be uttered to describe objects in the same category. By taking this relationship into account, our proposed method enables robots to acquire a more accurate language model and object concepts online. Experimental results verify this.

show abstract

Section: Related Workmentioning

confidence: 99%

Online Algorithm for Robots to Learn Object Concepts and Language Model

Nishihara

Nagaoka

Nagai

2017

IEEE Trans. Cogn. Dev. Syst.

View full text Add to dashboard Cite

show abstract

“…In [50], the authors use non-negative matrix factorization to learn a joint representation of gestures and spoken words. They show that the learned representations can be used to retrieve one modality given the other (e.g.…”

Section: Multimodal Fusionmentioning

confidence: 99%

Deep unsupervised network for multimodal perception, representation and classification

Droniou

Ivaldi

Sigaud

2015

Robotics and Autonomous Systems

View full text Add to dashboard Cite

In this paper, we tackle the problem of multimodal learning for autonomous robots. Autonomous robots interacting with humans in an evolving environment need the ability to acquire knowledge from their multiple perceptual channels in an unsupervised way. Most of the approaches in the literature exploit engineered methods to process each perceptual modality. In contrast, robots should be able to acquire their own features from the raw sensors, leveraging the information elicited by interaction with their environment: learning from their sensorimotor experience would result in a more efficient strategy in a life-long perspective. To this end, we propose an architecture based on deep networks, which is used by the humanoid robot iCub to learn a task from multiple perceptual modalities (proprioception, vision, audition). By structuring high-dimensional, multimodal information into a set of distinct sub-manifolds in a fully unsupervised way, it performs a substantial dimensionality reduction by providing both a symbolic representation of data and a fine discrimination between two similar stimuli. Moreover, the proposed network is able to exploit multimodal correlations to improve the representation of each modality alone.

show abstract

“…Many researchers have investigated methods for robots to learn relationships among multimodal sensory data [10], [11], [12], [13], [14]. The purpose of these studies was to predict one modality from another by using models.…”

Section: Introductionmentioning

confidence: 99%

Concept formation by robots using an infinite mixture of models

Nagaoka

Yoshiki

Nagai

et al. 2015

2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

We propose a method for a robot to form various concepts. The robot uses its embodiment to obtain visual, auditory, and haptic information by grasping, shaking, and observing objects. At the same time, a user teaches the robot object features through speech. From these kinds of information, the robot can form object concepts. The information obtained by the robot is converted into Bag-of-Words representations, which are classified into categories. The proposed method is based on a stochastic model, and objects can be classified by estimating their parameters. We introduce the Chinese restaurant process into the multimodal hierarchical Dirichlet process. This model is an infinite mixture of models and enables the robot to form various concepts such as object type, color, and so on. Therefore, the robot can form not only object concepts but also various concepts that are not represented by object concepts (e.g., color). Also, because the proposed method is based on a stochastic model, it makes it possible for the robot to estimate category and unobserved information of unseen objects. We implement the proposed method on a robot and show that it can form various concepts and perform various estimations for unseen objects.

show abstract

Learning semantic components from subsymbolic multimodal perception

Cited by 19 publications

References 17 publications

Online Algorithm for Robots to Learn Object Concepts and Language Model

Online Algorithm for Robots to Learn Object Concepts and Language Model

Deep unsupervised network for multimodal perception, representation and classification

Concept formation by robots using an infinite mixture of models

Contact Info

Product

Resources

About