Compositional properties of emergent languages in deep learning

Abstract

Recent findings in multi-agent deep learning systems point towards the emergence of compositional languages. These claims are often made without exact analysis or testing of the language.In this work, we analyze the emergent language resulting from two different cooperative multiagent game with more exact measures for compositionality. Our findings suggest that solutions found by deep learning models are often lacking the ability to reason on an abstract level therefore failing to generalize the learned knowledge to out of the training distribution examples. Strategies for testing compositional capacities and emergence of human-level concepts are discussed.

PatchGame: Learning to Signal Mid-level Patches in Referential Games

“…Existing approaches use the successful completion of the task or the correlation between learned language and semantic labels as evaluation metrics. Lowe et al [50] and Keresztury and Bruni [40] show that simple task success might not be a good or sufficient metric for evaluating the success of a game. They discuss heuristics and advocate measuring both positive signaling and positive listening independently to evaluate agents' communication.…”