The eyes grasp, the hands see: Metric category knowledge transfers between vision and touch

Abstract

Categorization of seen objects is often determined by the shapes of objects. However, shape is not exclusive to the visual modality: The haptic system also is expert at identifying shapes. Hence, an important question for understanding shape processing is whether humans store separate modality-dependent shape representations, or whether information is integrated into one multisensory representation. To answer this question, we created a metric space of computer-generated novel objects varying in shape. These objects were then printed using a 3-D printer, to generate tangible stimuli. In a categorization experiment, participants first explored the objects visually and haptically. We found that both modalities led to highly similar categorization behavior. Next, participants were trained either visually or haptically on shape categories within the metric space. As expected, visual training increased visual performance, and haptic training increased haptic performance. Importantly, however, we found that visual training also improved haptic performance, and vice versa. Two additional experiments showed that the location of the categorical boundary in the metric space also transferred across modalities, as did heightened discriminability of objects adjacent to the boundary. This observed transfer of metric category knowledge across modalities indicates that visual and haptic forms of shape information are integrated into a shared multisensory representation.

“Can touch this”: Cross-modal shape categorization performance is associated with microstructural characteristics of white matter association pathways

“…Participants on average needed 10.4 (SD = 5.3) blocks to finish the visual training—this is consistent with the data from Wallraven et al [, not reported] in which participants needed 8.5 blocks (SD = 5.9). Only 3 out of 37 participants were not able to finish within 20 blocks—during the final blocks, however, performance for these participants hovered around 92%, showing that they made only one or two errors during last two blocks (16 objects).…”

“…Average haptic accuracy was 80.5% (SD = 12.4%) ‐ again, this average performance is consistent with a re‐analysis from data of Wallraven et al [, not reported] in which participants achieved 84.5% (SD = 12.9%). Additionally, accuracy scores were normally distributed ( W = 0.94, P > 0.05).…”

“…The similarity between vision and touch also extends to categorization tasks with novel objects [Wallraven et al, ] and to real‐world natural objects [Gaißert and Wallraven, ]. In a recent study, we have shown that metric shape information trained in one modality (either vision or touch) symmetrically transfers to the other, untrained modality using a novel shape categorization task [Wallraven et al, ].…”

“…While these results—and those from other studies—speak strongly in favor of shared representations between vision and touch, one aspect that has so far not been studied in detail is the individual variability in this task. Indeed, in the study by Wallraven et al [], all participants showed proper training effects—the magnitude of these effects, however, differed from participant to participant and with that also participants' performance and ability to transfer information from vision to touch (and vice versa). In other words, there are people who are naturally better at cross‐modal information transfer than others.…”

Learning multisensory representations for auditory-visual transfer of sequence category knowledge: a probabilistic language of thought approach

“…This fact is demonstrated by cross-modal transfer of knowledge. If a person is trained to visually categorize a set of objects, this person will often be able to categorize novel objects from the same categories when objects are grasped but not seen (Wallraven et al, 2014;Yildirim & Jacobs, 2013). Because knowledge acquired during visual learning is used during haptic testing, this finding suggests the existence of both visual and haptic representations of objects.…”