Cross-modal visuo-haptic mental rotation: comparing objects between senses

Abstract

The simple experience of a coherent percept while looking and touching an object conceals an intriguing issue: different senses encode and compare information in different modality-specific reference frames. We addressed this problem in a cross-modal visuo-haptic mental rotation task. Two objects in various orientations were presented at the same spatial location, one visually and one haptically. Participants had to identify the objects as same or different. The relative angle between viewing direction and hand orientation was manipulated (Aligned versus Orthogonal). In an additional condition (Delay), a temporal delay was introduced between haptic and visual explorations while the viewing direction and the hand orientation were orthogonal to each other. Whereas the phase shift of the response time function was close to 0° in the Aligned condition, we observed a consistent phase shift in the hand’s direction in the Orthogonal condition. A phase shift, although reduced, was also found in the Delay condition. Counterintuitively, these results mean that seen and touched objects do not need to be physically aligned for optimal performance to occur. The present results suggest that the information about an object is acquired in separate visual and hand-centered reference frames, which directly influence each other and which combine in a time-dependent manner.

Rotation-independent representations for haptic movements

“…Mental rotation effect has been reported with haptic objects83637 and mutual interference between haptic and visual stimulation has also been reported2243. These studies suggest a common process for haptic and visual object representations.…”

“…1). Although similar effects of rotation angle have been reported for visual and haptic object representations83637, and the contribution of parietal and premotor cortices to mental rotation of visual stimuli has been suggested38, investigation of movement representations is a different issue. No study has compared the effect of rotation angle on reproducing movement patterns obtained through visual and haptic inputs, although stimulus objects have often been explored with haptic movements223139.…”

An Investigation into the Cause of Orientation-Sensitivity in Haptic Object Recognition

“…It should also benefit crossmodal processing (trying to recognise an object by touch after having seen it, and vice versa) and, probably more importantly, multisensory processing (Takahashi et al, 2009). However, fundamental differences in information acquisition for vision versus haptics and in the reference frames used by the two modalities may make it difficult for them to use common processes and representations (Volcic et al, 2010). This issue has been investigated from a variety of perspectives, for example, using imaging (e.g., Amedi et al, 2005;Lacey et al, 2009), multidimensional scaling of similarity ratings (Cooke et al, 2007;Gaissert et al, 2010) and psychophysical experiments (e.g., Craddock and Lawson, 2009;Lawson, 2009;Newell et al, 2001).…”

Spatial Memory for Patterns of Taps on the Fingers

“…99]. Given the correlation between touch and vision for spatial memory performance reported here and visuo-haptic interactions reported elsewhere, e.g., [21], it is interesting to consider the relative contributions of touch and vision to the development of automaticity in everyday activities, such as texting, playing a piano or a video game, or the remote operation of drones. Furthermore, basic cognitive research can be employed to evaluate the capacity for haptic working memory and our ability to shift attention from one modality to another.…”

“…However, the nature of this interaction is not obvious; using a crossmodal visuo-haptic spatial rotation task, [21] found that seen or touched objects do not need to be physically aligned to achieve optimal object-identification performance, yet the visual and haptic reference frames nevertheless directly influence each other. Underscoring the importance of reciprocal cross-modal interactions between touch and vision, motion aftereffects induced in the touch modality can influence the perception of analog motion aftereffects in the visual mode [22], a phenomenon that also works reciprocally.…”