Multisensory recognition of actively explored objects.

Abstract

Shape recognition can be achieved through vision or touch, raising the issue of how this information is shared across modalities. Here we provide a short review of previous findings on cross-modal object recognition and we provide new empirical data on multisensory recognition of actively explored objects. It was previously shown that, similar to vision, haptic recognition of objects fixed in space is orientation specific and that cross-modal object recognition performance was relatively efficient when these views of the objects were matched across the sensory modalities (Newell, Ernst, Tjan, & Bülthoff, 2001). For actively explored (i.e., spatially unconstrained) objects, we now found a cost in cross-modal relative to within-modal recognition performance. At first, this may seem to be in contrast to findings by Newell et al. (2001). However, a detailed video analysis of the visual and haptic exploration behaviour during learning and recognition revealed that one view of the objects was predominantly explored relative to all others. Thus, active visual and haptic exploration is not balanced across object views. The cost in recognition performance across modalities for actively explored objects could be attributed to the fact that the predominantly learned object view was not appropriately matched between learning and recognition test in the cross-modal conditions. Thus, it seems that participants naturally adopt an exploration strategy during visual and haptic object learning that involves constraining the orientation of the objects. Although this strategy ensures good within-modal performance, it is not optimal for achieving the best recognition performance across modalities.

From CT scanning to 3-D printing technology for the preoperative planning in laparoscopic splenectomy

“…Vision and touch generate functionally overlapping, but not necessarily equivalent, representations of 3-D shape. Experimental evidence suggests that haptic exploration can influence and improve visual perception of 3-D shape [16,17].…”

Cross-modal visuo-haptic mental rotation: comparing objects between senses

“…2007, 2009; Ernst et al. 2007). On the other hand, in a sequential matching task, an object presented in one modality was shortly afterwards compared with a test object through the other modality (Lawson 2009).…”

“…2001; Ernst et al. 2007; Lawson 2009), whereas in other cases the performance was unaffected irrespective of the change in orientation (Lacey et al. 2007, 2009; Lawson 2009).…”

Spatial Cognition Through the Keyhole: How Studying a Real‐World Domain Can Inform Basic Science—and Vice Versa

“…But we can use the hands to explore around structures. In fact, the hands are especially well suited to “seeing” the backs of objects and gathering 3D shape information (Ernst, Lange, & Newell, 2007; Klatzky, Lederman, & Reed, 1987; Newell, Ernst, Tjan, & Bultoff, 2001). Mechanoreceptors sense pressure across the fingers and palm.…”