Crossmodal Association of Visual and Haptic Material Properties of Objects in the Monkey Ventral Visual Cortex

Abstract

Just by looking at an object, we can recognize its non-visual properties, such as hardness. The visual recognition of non-visual object properties is generally accurate [1], and influences actions toward the object [2]. Recent studies suggest that, in the primate brain, this may involve the ventral visual cortex, which represents objects in a way that reflects not only visual but also non-visual object properties, such as haptic roughness, hardness, and weight [3-7]. This new insight raises a fundamental question: how does the visual cortex come to represent non-visual properties--knowledge that cannot be acquired directly through vision? Here we addressed this unresolved question using fMRI in macaque monkeys. Specifically, we explored whether and how simple visuo-haptic experience--just seeing and touching objects made of various materials--can shape representational content in the visual cortex. We measured brain activity evoked by viewing images of objects before and after the monkeys acquired the visuo-haptic experience and decoded the representational space from the activity patterns [8]. We show that simple long-term visuo-haptic experience greatly impacts representation in the posterior inferior temporal cortex, the higher ventral visual cortex. After the experience, but not before, the activity pattern in this region well reflected the haptic material properties of the experienced objects. Our results suggest that neural representation of non-visual object properties in the visual cortex emerges through long-term crossmodal exposure to objects. This highlights the importance of unsupervised learning of crossmodal associations through everyday experience [9-12] for shaping representation in the visual cortex.

Predicting precision grip grasp locations on three-dimensional objects

“…Both regions exhibit strong connections with AIP, which has been shown to represent the shape, size, and orientation of 3D objects, as well as the shape of the handgrip, grip size, and hand-orientation(46). Additionally, visual material properties, including object weight, are thought to be encoded in the ventral visual cortex(47–51), and it has been suggested that AIP might play a unique role in linking components of the ventral visual stream involved in object recognition to hand motor system(52). Therefore, the neural circuit formed between F5, F2, and particularly AIP is a strong candidate for combining the multifaceted components of visually guided grasping identified in this work(53–57).…”

Image Statistics and the Representation of Material Properties in the Visual Cortex

“…Obviously, there have to be more elaborate computations that can later judge materials in a more fine-grained manner, carried out in higher visual areas. Such computations could, for example, take 3-dimensional information into account to determine a surface’s roughness, or access haptic representations of a surface ( Goda et al, 2016 ). So, even though low-level statistics, namely pixel statistics, filter parameters and spatial frequency information seem to play a role in material perception, mid- and high-level areas of the visual system must be involved in the perception of material properties.…”

“… Baumgartner et al (2013) found visual and haptic representations to be tightly linked. This is likely to be the result of learning processes ( Goda et al, 2016 ). Therefore it is difficult to attribute certain properties to one of the senses but from our earlier work it seems that texturedness can be judged more reliably in the visual sense, while roughness is a property that is easily and reliably accessible to both senses from an object’s material surface.…”

Crossmodal association of auditory and visual material properties in infants

“…The neural basis of material category perception has been investigated in humans 14 , 15 and monkeys 2 , 3 . Functional magnetic resonance imaging (fMRI) studies revealed cortical activation in the ventral visual pathway in relation to visual-based material perception; the fusiform gyrus in humans 15 and the inferior temporal cortex in monkeys 2 , 3 . On the other hand, the activation in the ventro-medial pathway was demonstrated in relation to auditory-based material perception in humans 14 .…”

“…On the other hand, the activation in the ventro-medial pathway was demonstrated in relation to auditory-based material perception in humans 14 . However, these studies focused on the high-order visual areas, where material information could be processed progressively 2 . Goda et al .…”