Grounding of Word Meanings in Latent Dirichlet Allocation-Based Multimodal Concepts

Nagaoka, Tomoaki; Araki, Takaya; Nagai, Takayuki; Iwahashi, Naoto

doi:10.1163/016918611x595035

Cited by 35 publications

(23 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their method allowed the robot to acquire phonemes and words from visual and auditory information through interaction with the human. Nakamura et al ( 2011a , b ) proposed multimodal latent Dirichlet allocation (MLDA) and a multimodal hierarchical Dirichlet process (MHDP) that enables the categorization of objects from multimodal information, i.e., visual, auditory, haptic, and word information. Their methods enabled more accurate object categorization by using multimodal information.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots

Taniguchi

Cangelosi

2017

Front. Neurorobot.

View full text Add to dashboard Cite

In this paper, we propose a Bayesian generative model that can form multiple categories based on each sensory-channel and can associate words with any of the four sensory-channels (action, position, object, and color). This paper focuses on cross-situational learning using the co-occurrence between words and information of sensory-channels in complex situations rather than conventional situations of cross-situational learning. We conducted a learning scenario using a simulator and a real humanoid iCub robot. In the scenario, a human tutor provided a sentence that describes an object of visual attention and an accompanying action to the robot. The scenario was set as follows: the number of words per sensory-channel was three or four, and the number of trials for learning was 20 and 40 for the simulator and 25 and 40 for the real robot. The experimental results showed that the proposed method was able to estimate the multiple categorizations and to learn the relationships between multiple sensory-channels and words accurately. In addition, we conducted an action generation task and an action description task based on word meanings learned in the cross-situational learning scenario. The experimental results showed that the robot could successfully use the word meanings learned by using the proposed method.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Attamimi et al ( 2016 ) proposed multilayered MLDA (mMLDA) that hierarchically integrates multiple MLDAs as an extension of Nakamura et al ( 2011a ). They performed an estimation of the relationships among words and multiple concepts by weighting the learned words according to their mutual information as a post-processing step.…”

Section: Related Workmentioning

confidence: 99%

Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots

Taniguchi

Cangelosi

2017

Front. Neurorobot.

View full text Add to dashboard Cite

show abstract

“…Qu & Chai's method based on the IBM translation model [12] estimates the word-entity association probability. Nakamura et al proposed a method to learn object concepts and word meanings from multimodal information and verbal information [10]. The method proposed in [10] is a categorization method based on multimodal latent Dirichlet allocation (MLDA) that enables the acquisition of object concepts from multimodal information, such as visual, auditory, and haptic information [13].…”

Section: A Lexical Acquisitionmentioning

confidence: 99%

“…Nakamura et al proposed a method to learn object concepts and word meanings from multimodal information and verbal information [10]. The method proposed in [10] is a categorization method based on multimodal latent Dirichlet allocation (MLDA) that enables the acquisition of object concepts from multimodal information, such as visual, auditory, and haptic information [13]. Araki et al addressed the development of a method combining unsupervised word segmentation from uttered sentences by a nested Pitman-Yor language model (NPYLM) [14] and the learning of object concepts by MLDA [11].…”

Section: A Lexical Acquisitionmentioning

confidence: 99%

Spatial Concept Acquisition for a Mobile Robot that Integrates Self-Localization and Unsupervised Word Discovery from Spoken Sentences

Taniguchi

Inamura

2016

IEEE Trans. Cogn. Dev. Syst.

View full text Add to dashboard Cite

Abstract-In this paper, we propose a novel unsupervised learning method for the lexical acquisition of words related to places visited by robots, from human continuous speech signals. We address the problem of learning novel words by a robot that has no prior knowledge of these words except for a primitive acoustic model. Further, we propose a method that allows a robot to effectively use the learned words and their meanings for self-localization tasks. The proposed method is nonparametric Bayesian spatial concept acquisition method (SpCoA) that integrates the generative model for self-localization and the unsupervised word segmentation in uttered sentences via latent variables related to the spatial concept. We implemented the proposed method SpCoA on SIGVerse, which is a simulation environment, and TurtleBot 2, which is a mobile robot in a real environment. Further, we conducted experiments for evaluating the performance of SpCoA. The experimental results showed that SpCoA enabled the robot to acquire the names of places from speech sentences. They also revealed that the robot could effectively utilize the acquired spatial concepts and reduce the uncertainty in self-localization.

show abstract

“…Yu and Ballard (2004) explored a framework that learns the association between objects and their spoken names in day-to-day tasks. Nakamura et al (2011) introduced a multimodal categorization applied to robotics. Their framework exploited the relation of concepts in different modalities (visual, audio and haptic) using a Multimodal latent Dirichlet allocation.…”

Section: Introductionmentioning

confidence: 99%

Symbol Grounding Association in Multimodal Sequences with Missing Elements

Raue

Dengel

Breuel

et al. 2018

jair

View full text Add to dashboard Cite

In this paper, we extend a symbolic association framework for being able to handle missing elements in multimodal sequences. The general scope of the work is the symbolic associations of object-word mappings as it happens in language development in infants. In other words, two different representations of the same abstract concepts can associate in both directions. This scenario has been long interested in Artificial Intelligence, Psychology, and Neuroscience. In this work, we extend a recent approach for multimodal sequences (visual and audio) to also cope with missing elements in one or both modalities. Our method uses two parallel Long Short-Term Memories (LSTMs) with a learning rule based on EM-algorithm. It aligns both LSTM outputs via Dynamic Time Warping (DTW). We propose to include an extra step for the combination with the max operation for exploiting the common elements between both sequences. The motivation behind is that the combination acts as a condition selector for choosing the best representation from both LSTMs. We evaluated the proposed extension in the following scenarios: missing elements in one modality (visual or audio) and missing elements in both modalities (visual and sound). The performance of our extension reaches better results than the original model and similar results to individual LSTM trained in each modality.

show abstract

Grounding of Word Meanings in Latent Dirichlet Allocation-Based Multimodal Concepts

Cited by 35 publications

References 9 publications

Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots

Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots

Spatial Concept Acquisition for a Mobile Robot that Integrates Self-Localization and Unsupervised Word Discovery from Spoken Sentences

Symbol Grounding Association in Multimodal Sequences with Missing Elements

Contact Info

Product

Resources

About