Acceleration and deceleration at constant speed: systematic modulation of motion perception by kinematic sonification

Abstract

Many domains of human behavior are based on multisensory representations. Knowledge about the principles of multisensory integration is useful to configure real-time movement information for the online support of perceptuomotor processes (motor perception, control, and learning). A powerful method for generating real-time information is movement sonification. Remarkable evidence exists on movement-acoustic real-time information being effective in behavioral domains (music training, motor rehabilitation, sports). Here, we investigate whether and how biological motion perception can be enhanced, substituted, or modulated by kinematic sonification, with a focus on pitch coding. We work with gross motor cyclic movements and investigate the effectiveness of pitch scaling and consistent transposition on audiovisual motor perception accuracy (Experiment A). Beyond that, a new kind of audiovisual stimulus with inconsistent pitch transposition is used to produce a directed modulation of the integrated audiovisual percept (Experiment B). Results from Experiment A indicate pitch being powerful for mediating kinematic information to enhance motor perception and substituting information between perceptual modalities, even exceeding visual performance. Beyond these findings, results from Experiment B indicate that visual estimations of movement velocity can be enhanced or reduced auditorily. Movement sonification used for reshaping intermodal adjustments should be a powerful new tool for subconsciously shaping human movement patterns in the future.

Auditory Coding of Reaching Space

“…Kinematic movement sonification can provide information for perception and action for humans, being classified as musical. This result corresponds to other studies with movement sonification [8,14,27,55].…”

Training proprioception with sound: effects of real‐time auditory feedback on intermodal learning

“…According to Effenberg et al, 8 convergent sensory feedback, which shares a high level of spatiotemporal proximity, can get implicitly fused to promote intermodal learning (in this case auditory and proprioceptive). 10 Here, an additional inference can be drawn from literature emphasizing the importance of intermodal knowledge for obtaining spatial knowledge of the body in space. 52,53 Likewise, evidence from neuroimaging studies also supports the notion that a high level of stimulus-response consistency (meaningful organization of perceptual and motion events) can promote sensorimotor coactivations 27 and motor priming.…”

“…3,8 Research conclusively suggests that mapping a performer's action with real-time auditory feedback can enhance both the perceptuomotor representations in the brain and motor performance. [8][9][10][11][12] Strong influence of real-time auditory feedback on motor performance was thought to be due to its influence over the proprioceptive modality. [13][14][15][16] Hasegawa et al, 16 for instance, reported that training with auditory augmented biofeedback might facilitate the integration of auditory and proprioceptive systems.…”

“…Research conclusively suggests that mapping a performer's action with real‐time auditory feedback can enhance both the perceptuomotor representations in the brain and motor performance . Strong influence of real‐time auditory feedback on motor performance was thought to be due to its influence over the proprioceptive modality .…”

Modulation of ellipses drawing by sonification

“…Interestingly, auditory system reacts faster than the visual system and its proper frequency is well adapted to provide augmented information during movement execution (Vinken et al 2013). In particular, the use of real-time continuous auditory feedback directly related to movement parameters is called "movement sonification" (Bevilacqua et al 2016;Dubus and Bresin 2013). In such systems, initially developed for artistic purpose, the sound is continuously modified by the movements of the human performer.…”