Emergence of Number Sense in Deep Multi-modal Neural Networks

Abstract

Associating multimodal information is essential for human cognitive abilities, and its failure results in a wide range of neuropsychological symptoms. Multimodal learning has attracted attention in the field of machine learning. Particularly, investigating the impact of multimodal learning on the representation of information could facilitate understanding human multimodal association learning. Herein, we used a multimodal deep learning model as a computational model for multimodal association in the brain. We compared the representations of numerical information, i.e., hand-written digits and images of geometric figures, learned in single- and multi-modal ways, which presumably corresponded to the human cognitive process of learning number sense. The multimodal training produced better latent representation in terms of clustering quality, consistent with previous findings of multimodal learning in deep neural networks. Moreover, learned representation using the model exhibited superior performance in the downstream arithmetic task. Our novel findings experimentally demonstrated that changes in acquired latent representations in multimodal association learning were directly related to cognitive functions. This supports the possibility of multimodal learning research offering novel insights into understanding higher cognitive functions in humans, including mathematical skills.