Emergent communication through Metropolis-Hastings naming game with deep generative models

Taniguchi, Tadahiro; Yoshida, Yuto; Matsui, Yuta; Le Hoang, Nguyen; Taniguchi, Akira; Hagiwara, Yoshinobu

doi:10.1080/01691864.2023.2260856

Cited by 10 publications

(28 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our primary objective was to enable agents to process multimodal data from various sources, such as images, text, and audio, and to integrate this information into a cohesive representation. Building on the foundation of Inter-GMM + VAE (Taniguchi et al, 2023), we obtain the following results:…”

Section: Discussionmentioning

confidence: 99%

“…The study by Taniguchi et al (2023) provided a comprehensive explanation and validation of the approach, demonstrating its guaranteed convergence as an approximate decentralized Bayesian inference of shared representations…”

Section: Metropolis-hastings Naming Gamementioning

confidence: 99%

“…Agent B, as a listener, (3) evaluates whether to accept this sign based on the acceptance rate r, then (4) updates its own parameters through a sampling process. Next, two agents swap their roles: Agent B speaks, and Agent A listens (Taniguchi et al, 2023) posterior q ϕ (z|x). The training process involves optimizing the Evidence Lower BOund (ELBO):…”

Section: Variational Autoencoder (Vae) and Multimodal Vae (Mvae)mentioning

confidence: 99%

“…In the context of symbol emergence systems that employ DGM with multimodal data, three critical questions emerge that are yet to be addressed in previous works (Hagiwara et al, 2019;2022;Taniguchi et al, 2023):…”

Section: Introductionmentioning

confidence: 99%

“…The MH naming game employs a unique probability-based approach to evaluate the acceptance and rejection of information during agent interactions based on the judgment ratio calculated using the MH algorithm ( Hastings, 1970 ). By focusing on joint attention and incorporating acceptance probabilities, the agents can improve their abilities to exchange information and form shared signs or vocabularies ( Taniguchi et al, 2023 ).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Emergent communication of multimodal deep generative models based on Metropolis-Hastings naming game

Hoang,

Taniguchi,

Hagiwara

et al. 2024

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

Deep generative models (DGM) are increasingly employed in emergent communication systems. However, their application in multimodal data contexts is limited. This study proposes a novel model that combines multimodal DGM with the Metropolis-Hastings (MH) naming game, enabling two agents to focus jointly on a shared subject and develop common vocabularies. The model proves that it can handle multimodal data, even in cases of missing modalities. Integrating the MH naming game with multimodal variational autoencoders (VAE) allows agents to form perceptual categories and exchange signs within multimodal contexts. Moreover, fine-tuning the weight ratio to favor a modality that the model could learn and categorize more readily improved communication. Our evaluation of three multimodal approaches - mixture-of-experts (MoE), product-of-experts (PoE), and mixture-of-product-of-experts (MoPoE)–suggests an impact on the creation of latent spaces, the internal representations of agents. Our results from experiments with the MNIST + SVHN and Multimodal165 datasets indicate that combining the Gaussian mixture model (GMM), PoE multimodal VAE, and MH naming game substantially improved information sharing, knowledge formation, and data reconstruction.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Metropolis-hastings Naming Gamementioning

confidence: 99%

Section: Variational Autoencoder (Vae) and Multimodal Vae (Mvae)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations