Neuro-SERKET: Development of Integrative Cognitive System Through the Composition of Deep Probabilistic Generative Models

Taniguchi, Tadahiro; Nagaoka, Tomoaki; Suzuki, Masahiro; Kuniyasu, Ryo; Hayashi, Kaede; Taniguchi, Akira; Horii, Takato; Nagai, Takayuki

doi:10.1007/s00354-019-00084-w

Cited by 45 publications

(36 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The studies on cognitive architectures for integrating semiotic and multimodal information are also critical for achieving lifelong learning in each local service environment. We believe that SERKET and probabilistic generative modelbased approaches are promising [75][76][77].…”

Section: Discussionmentioning

confidence: 99%

Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society

et al. 2021

Self Cite

View full text Add to dashboard Cite

The installation of remotely-operated service robots in the environments of our daily life (including offices, homes, and hospitals) can improve work-from-home policies and enhance the quality of the so-called new normal. However, it is evident that remotely-operated robots must have partial autonomy and the capability to learn and use local semiotic knowledge. In this paper, we argue that the development of semiotically adaptive cognitive systems is key to the installation of service robotics technologies in our service environments. To achieve this goal, we describe three challenges: the learning of local knowledge, the acceleration of onsite and online learning, and the augmentation of human-robot interactions.

show abstract

Section: Discussionmentioning

confidence: 99%

Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…This extension is implemented by applying a simultaneous learning model of words and concepts [20] and the multi-channel categorization model [38] to Inter-MDM. (2) Extension of Inter-MDM to a model that can generate images directly by applying the deep generative model [39], the neural topic model [40], and Neuro-SERKET [41]. (3) Extension of Inter-MDM from two agents to a naming game involving three or more agents.…”

Section: Discussionmentioning

confidence: 99%

Multiagent Multimodal Categorization for Symbol Emergence: Emergent Communication via Interpersonal Cross-modal Inference

Hagiwara¹,

Furukawa²,

Taniguchi³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper describes a computational model of multiagent multimodal categorization that realizes emergent communication. We clarify whether the computational model can reproduce the following functions in a symbol emergence system, comprising two agents with different sensory modalities playing a naming game. (1) Function for forming a shared lexical system that comprises perceptual categories and corresponding signs, formed by agents through individual learning and semiotic communication between agents. (2) Function to improve the categorization accuracy in an agent via semiotic communication with another agent, even when some sensory modalities of each agent are missing.(3) Function that an agent infers unobserved sensory information based on a sign sampled from another agent in the same manner as cross-modal inference. We propose an interpersonal multimodal Dirichlet mixture (Inter-MDM), which is derived by dividing an integrative probabilistic generative model, which is obtained by integrating two Dirichlet mixtures (DMs). The Markov chain Monte Carlo algorithm realizes emergent communication. The experimental results demonstrated that Inter-MDM enables agents to form multimodal categories and appropriately share signs between agents. It is shown that emergent communication improves categorization accuracy, even when some sensory modalities are missing. Inter-MDM enables an agent to predict unobserved information based on a shared sign.

show abstract

“…Thus, a latent space that is suitable for clustering is learned by regularizing q ϕ (w v |o), using p (w v |w w , Θ) as a prior. We use the Symbol Emergence in Robotics tool KIT (SERKET) (Nakamura et al, 2018;Taniguchi et al, 2020) to implement the integrated model. SERKET is a framework for constructing a large-scale model composed of small models.…”

Section: Parameter Estimation Of Integrated Modelmentioning

confidence: 99%

Robot Concept Acquisition Based on Interaction Between Probabilistic and Deep Generative Models

Kuniyasu¹,

Nagaoka²,

Taniguchi³

et al. 2021

Front. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

We propose a method for multimodal concept formation. In this method, unsupervised multimodal clustering and cross-modal inference, as well as unsupervised representation learning, can be performed by integrating the multimodal latent Dirichlet allocation (MLDA)-based concept formation and variational autoencoder (VAE)-based feature extraction. Multimodal clustering, representation learning, and cross-modal inference are critical for robots to form multimodal concepts from sensory data. Various models have been proposed for concept formation. However, in previous studies, features were extracted using manually designed or pre-trained feature extractors and representation learning was not performed simultaneously. Moreover, the generative probabilities of the features extracted from the sensory data could be predicted, but the sensory data could not be predicted in the cross-modal inference. Therefore, a method that can perform clustering, feature learning, and cross-modal inference among multimodal sensory data is required for concept formation. To realize such a method, we extend the VAE to the multinomial VAE (MNVAE), the latent variables of which follow a multinomial distribution, and construct a model that integrates the MNVAE and MLDA. In the experiments, the multimodal information of the images and words acquired by a robot was classified using the integrated model. The results demonstrated that the integrated model can classify the multimodal information as accurately as the previous model despite the feature extractor learning in an unsupervised manner, suitable image features for clustering can be learned, and cross-modal inference from the words to images is possible.

show abstract

Neuro-SERKET: Development of Integrative Cognitive System Through the Composition of Deep Probabilistic Generative Models

Cited by 45 publications

References 36 publications

Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society

Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society

Multiagent Multimodal Categorization for Symbol Emergence: Emergent Communication via Interpersonal Cross-modal Inference

Robot Concept Acquisition Based on Interaction Between Probabilistic and Deep Generative Models

Contact Info

Product

Resources

About