Vision and Touch: Multiple or Multisensory Representations of Objects?

Abstract

The relationship between visually and haptically derived representations of objects is an important question in multisensory processing and, increasingly, in mental representation. We review evidence for the format and properties of these representations, and address possible theoretical models. We explore the relevance of visual imagery processes and highlight areas for further research, including the neglected question of asymmetric performance in the visuo-haptic cross-modal memory paradigm. We conclude that the weight of evidence suggests the existence of a multisensory representation, spatial in format, and flexibly accessible by both bottom-up and top-down inputs, although efficient comparison between modality-specific representations cannot entirely be ruled out.

Cross-modal object recognition and dynamic weighting of sensory inputs in a fish

“…To recognize objects cross-modally, information about characteristic object features has to be stored in some kind of neuronal representation, which is subsequently accessible by the other sense. This could be achieved either through a comparison of modality-specific representations or through a single multimodal representation stored in a multisensory brain area (9,32,33). In mammals, cross-modal object recognition is associated with cortical structures such as the prefrontal cortex (6), the perirhinal, or the posterior parietal cortex (5, 7).…”

Haptic perception: A tutorial

“…For example, in the study by Kitada et al (2009), greater modalityindependent activation of the fusiform face area within the fusiform gyrus and of the extrastriate body area in the lateral occipital cortex was found when subjects identified faces or other body parts (hands, feet), respectively, as compared with nonbiological control objects (bottles). Lacey et al (2007) and Sathian and Lacey (2008) have now examined a large number of studies (including, but not limited to, most in this section) with respect to whether visual mediation and/or multisensory processing is used during tactile/haptic perception. Overall, they conclude that the current evidence collectively points to the creation of a multisensory spatial representation that may be flexibly accessed via either knowledge-driven or stimulus-driven processes.…”

“…Nonsignificant, or significant but very low, correlations were obtained between activation patterns produced by conditions in which subjects haptically explored versus visually imagined the objects, between the activation patterns of subjects who described themselves as using visual imagery in the haptic condition and those who did not, and between scores on the VVIQ questionnaire (Marks, 1973), which measures the vividness of subjects' visual imaging abilities, and activation during haptic object identification. As Kosslyn and Thompson (1993) and Lacey et al (2007) have noted, visual imagery is a highly complex process that consists of multiple components (e.g., image generation, maintenance, inspection, and transformation). Greater understanding of visual imagery, together with a more extensive battery of evaluation tasks, is much …”

“…Visual involvement could include (1) knowledge-directed processes (e.g., anticipatory visual imagery, visual memory) that may assist or mediate tactual performance; (2) stimulus-directed activation of visual cortical areas by tactual inputs, implying that these so-called "visual" areas are actually "multisensory"; or (3) both knowledge-driven and stimulus-driven processes (Lacey, Campbell, & Sathian, 2007;Sathian & Lacey, 2008).…”

A percepção tátil de variáveis gráficas no reconhecimento de objetos tridimensionais para cegos congênitos

“…Os corpos caracterizam-se como tridimensionais, e o ato de circunscrevê-los pressupõe a limitação física do todo. A superfície é dita como bidimensional percebida no espaço tridimensional, o que acarreta na sua representação como a localização no espaço (Lacey, S, Campbell, C, & Sathian, K, 2007). Embora retratada como bidimensional, no mundo físico do homem e de seus artefatos diz-se predominantemente bidimensional e percebida em um espaço tridimensional em sua manipulação (Lacey, S, Campbell, C & Sathian, K, 2007).…”