Symbol Emergence as an Interpersonal Multimodal Categorization

Hagiwara, Yoshinobu; Kobayashi, Hiroyoshi; Taniguchi, Akira; Taniguchi, Tadahiro

doi:10.3389/frobt.2019.00134

Cited by 23 publications

(45 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…multimodal concept formation) have a strong relationship [1,64,65]. 3 Figure 2 depicts the overview of a symbol emergence system (SES) which is the core concept of symbol emergence in robotics [64,71]. The theory of SES argues that semiotic communication should be based on embodied environmental adaptation.…”

Section: Discussionmentioning

confidence: 99%

Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society

et al. 2021

Self Cite

View full text Add to dashboard Cite

The installation of remotely-operated service robots in the environments of our daily life (including offices, homes, and hospitals) can improve work-from-home policies and enhance the quality of the so-called new normal. However, it is evident that remotely-operated robots must have partial autonomy and the capability to learn and use local semiotic knowledge. In this paper, we argue that the development of semiotically adaptive cognitive systems is key to the installation of service robotics technologies in our service environments. To achieve this goal, we describe three challenges: the learning of local knowledge, the acceleration of onsite and online learning, and the augmentation of human-robot interactions.

show abstract

Section: Discussionmentioning

confidence: 99%

Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Regarding symbol emergence systems based on multimodal sensory information, the following three questions arise that have not been verified in previous works [5][6][7]:…”

Section: Introductionmentioning

confidence: 89%

Section: Introductionmentioning

confidence: 99%

“…Hagiwara et al proposed a computational model of a symbol emergence system comprising two agents that perform categorization based on a visual modality, i.e., a single modality [5]. We call the model proposed in [5] interpersonal Dirichlet mixture (Inter-DM) in this study because the model is obtained by combining two Dirichlet mixtures (DMs). Inter-DM is an advanced version of the Talking Heads experiment, in which various computational models of language emergence using perceptual categories based on sensory experiences were proposed by Steels et al [6].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multiagent Multimodal Categorization for Symbol Emergence: Emergent Communication via Interpersonal Cross-modal Inference

Hagiwara¹,

Furukawa²,

Taniguchi³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper describes a computational model of multiagent multimodal categorization that realizes emergent communication. We clarify whether the computational model can reproduce the following functions in a symbol emergence system, comprising two agents with different sensory modalities playing a naming game. (1) Function for forming a shared lexical system that comprises perceptual categories and corresponding signs, formed by agents through individual learning and semiotic communication between agents. (2) Function to improve the categorization accuracy in an agent via semiotic communication with another agent, even when some sensory modalities of each agent are missing.(3) Function that an agent infers unobserved sensory information based on a sign sampled from another agent in the same manner as cross-modal inference. We propose an interpersonal multimodal Dirichlet mixture (Inter-MDM), which is derived by dividing an integrative probabilistic generative model, which is obtained by integrating two Dirichlet mixtures (DMs). The Markov chain Monte Carlo algorithm realizes emergent communication. The experimental results demonstrated that Inter-MDM enables agents to form multimodal categories and appropriately share signs between agents. It is shown that emergent communication improves categorization accuracy, even when some sensory modalities are missing. Inter-MDM enables an agent to predict unobserved information based on a shared sign.

show abstract

“…Moreover, we consider that the integration of the MLDA and representation learning is also significant in terms of leveraging past research. Although pre-designed or pre-trained feature extractors have been used, various studies (Attamimi et al, 2014;Nakamura and Nagai, 2017;Hagiwara et al, 2019;Miyazawa et al, 2019) in addition to the above have revealed the effectiveness of the MLDA for multimodal concept formation. That is, the integration of the MLDA and representation learning makes it possible to develop these studies further.…”

Section: Introductionmentioning

confidence: 99%

Robot Concept Acquisition Based on Interaction Between Probabilistic and Deep Generative Models

Kuniyasu¹,

Nagaoka²,

Taniguchi³

et al. 2021

Front. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

We propose a method for multimodal concept formation. In this method, unsupervised multimodal clustering and cross-modal inference, as well as unsupervised representation learning, can be performed by integrating the multimodal latent Dirichlet allocation (MLDA)-based concept formation and variational autoencoder (VAE)-based feature extraction. Multimodal clustering, representation learning, and cross-modal inference are critical for robots to form multimodal concepts from sensory data. Various models have been proposed for concept formation. However, in previous studies, features were extracted using manually designed or pre-trained feature extractors and representation learning was not performed simultaneously. Moreover, the generative probabilities of the features extracted from the sensory data could be predicted, but the sensory data could not be predicted in the cross-modal inference. Therefore, a method that can perform clustering, feature learning, and cross-modal inference among multimodal sensory data is required for concept formation. To realize such a method, we extend the VAE to the multinomial VAE (MNVAE), the latent variables of which follow a multinomial distribution, and construct a model that integrates the MNVAE and MLDA. In the experiments, the multimodal information of the images and words acquired by a robot was classified using the integrated model. The results demonstrated that the integrated model can classify the multimodal information as accurately as the previous model despite the feature extractor learning in an unsupervised manner, suitable image features for clustering can be learned, and cross-modal inference from the words to images is possible.

show abstract

Symbol Emergence as an Interpersonal Multimodal Categorization

Cited by 23 publications

References 52 publications

Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society

Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society

Multiagent Multimodal Categorization for Symbol Emergence: Emergent Communication via Interpersonal Cross-modal Inference

Robot Concept Acquisition Based on Interaction Between Probabilistic and Deep Generative Models

Contact Info

Product

Resources

About