When Ears Drive Hands: The Influence of Contact Sound on Reaching to Grasp

Abstract

BackgroundMost research on the roles of auditory information and its interaction with vision has focused on perceptual performance. Little is known on the effects of sound cues on visually-guided hand movements.Methodology/Principal FindingsWe recorded the sound produced by the fingers upon contact as participants grasped stimulus objects which were covered with different materials. Then, in a further session the pre-recorded contact sounds were delivered to participants via headphones before or following the initiation of reach-to-grasp movements towards the stimulus objects. Reach-to-grasp movement kinematics were measured under the following conditions: (i) congruent, in which the presented contact sound and the contact sound elicited by the to-be-grasped stimulus corresponded; (ii) incongruent, in which the presented contact sound was different to that generated by the stimulus upon contact; (iii) control, in which a synthetic sound, not associated with a real event, was presented. Facilitation effects were found for congruent trials; interference effects were found for incongruent trials. In a second experiment, the upper and the lower parts of the stimulus were covered with different materials. The presented sound was always congruent with the material covering either the upper or the lower half of the stimulus. Participants consistently placed their fingers on the half of the stimulus that corresponded to the presented contact sound.Conclusions/SignificanceAltogether these findings offer a substantial contribution to the current debate about the type of object representations elicited by auditory stimuli and on the multisensory nature of the sensorimotor transformations underlying action.

Multisensory texture exploration at the tip of the pen

“…When comparing this work with other research that showed how texture sounds can affect motor behavior (Castiello et al, 2010), it should be considered that, differently from those experiments, here the sound (and vibratory) feedback signals have always been congruent with the visual stimulus. It is possible that, if playing with incongruent sound feedback, significant behavioral differences would have been registered.…”

“…Castiello et al (2010) showed that the sound of fingers on different material textures affects movement duration in a reaching-to-grasp task, with sounds that are congruent with the visual appearance producing shorter movement durations. Moreover, their experiments provided evidence that the contact sound is used by both the planning and the on-line control systems at the neural level.…”

The unity assumption facilitates cross-modal binding of musical, non-speech stimuli: The role of spectral and amplitude envelope cues

“…For these selfgenerated actions, there appears to be a predictive element to action-outcome temporal binding: as long as the probability of a sensory outcome is sufficiently high, the timing of a voluntary action will be pulled toward that expected outcome's position in time, even if the outcome does not occur (Moore & Haggard, 2008). In addition to influencing the perceived timing of actions and their outcomes, research has indicated that congruency between auditory and visual information can also influence the actual speed of motor actions (Castiello, Giordano, Begliomini, Ansuini, & Grassi, 2010).…”

The effect of real-time auditory feedback on learning new characters

“…sounds evoking a natural action on an object, were used (Aramaki & Kronland-Martinet, 2006;Conan et al, 2013;Thoret, Aramaki, Gondre, Kronland-Martinet, & Ystad, 2013). Indeed, the effectiveness of auditory FB strongly depends on the intuitive and correct interpretation of the applied mapping functions and metaphors (Castiello, Giordano, Begliomini, Ansuini, & Grassi, 2010).…”