A Rational Analysis of the Acquisition of Multisensory Representations

Abstract

How do people learn multisensory, or amodal, representations, and what consequences do these representations have for perceptual performance? We address this question by performing a rational analysis of the problem of learning multisensory representations. This analysis makes use of a Bayesian nonparametric model that acquires latent multisensory features that optimally explain the unisensory features arising in individual sensory modalities. The model qualitatively accounts for several important aspects of multisensory perception: (a) it integrates information from multiple sensory sources in such a way that it leads to superior performances in, for example, categorization tasks; (b) its performances suggest that multisensory training leads to better learning than unisensory training, even when testing is conducted in unisensory conditions; (c) its multisensory representations are modality invariant; and (d) it predicts ''missing'' sensory representations in modalities when the input to those modalities is absent. Our rational analysis indicates that all of these aspects emerge as part of the optimal solution to the problem of learning to represent complex multisensory environments.

Learning Multisensory Integration and Coordinate Transformation via Density Estimation

“…This allows for greater generality since it does not require foreknowledge of which populations should be forced to share information: if the information in the input populations is redundant, it will be “integrated” in the hidden units, and conversely. More recently, the idea of treating multisensory integration as a density estimation problem has been proposed independently by [53], a complementary report that explores both cognitive and neural implications of this view, without proposing an explicit neural implementation. As in [50], [52], then, no attempt is made to employ biological learning rules.…”

Discovering latent causes in reinforcement learning

“…The parameter l plays the same role as a, controlling how the number of latent causes grows as more customers enter, and hence the 'complexity' of the model. Soto and colleagues used the Indian buffet process in their latent-cause model of compound conditioning [13], and it has also been used in models of perceptual feature learning [58][59][60].…”

The origins and structure of quantitative concepts

“…To address the hypothesis that multisensory representations are tuned over time from unisensory inputs, Yildirim and Jacobs (2012) developed a nonparametric Bayesian model with multisensory feature variables (without specifying the number of features in advance) that are tuned in the course of learning. These multisensory feature variables are latent in the sense that they are unobserved: they must be estimated from the values of the observed unisensory inputs.…”

“…In one example, a recent type of neural network used for dimensionality reduction called deep networks (Hinton & Salakhudtinov, 2006) has successfully modeled neurophysiological and behavioral data from visual numerosity estimation and comparison tasks (Stoianov & Zorzi, 2012). However, like Yildirim and Jacobs’ model (2012), PCA and its more sophisticated variants have important limitations. For example, in classical PCA, the researcher must specify the number of principal components to find in advance of performing the computation.…”