Movement Sonification: Effects on Motor Learning beyond Rhythmic Adjustments

Abstract

Motor learning is based on motor perception and emergent perceptual-motor representations. A lot of behavioral research is related to single perceptual modalities but during last two decades the contribution of multimodal perception on motor behavior was discovered more and more. A growing number of studies indicates an enhanced impact of multimodal stimuli on motor perception, motor control and motor learning in terms of better precision and higher reliability of the related actions. Behavioral research is supported by neurophysiological data, revealing that multisensory integration supports motor control and learning. But the overwhelming part of both research lines is dedicated to basic research. Besides research in the domains of music, dance and motor rehabilitation, there is almost no evidence for enhanced effectiveness of multisensory information on learning of gross motor skills. To reduce this gap, movement sonification is used here in applied research on motor learning in sports. Based on the current knowledge on the multimodal organization of the perceptual system, we generate additional real-time movement information being suitable for integration with perceptual feedback streams of visual and proprioceptive modality. With ongoing training, synchronously processed auditory information should be initially integrated into the emerging internal models, enhancing the efficacy of motor learning. This is achieved by a direct mapping of kinematic and dynamic motion parameters to electronic sounds, resulting in continuous auditory and convergent audiovisual or audio-proprioceptive stimulus arrays. In sharp contrast to other approaches using acoustic information as error-feedback in motor learning settings, we try to generate additional movement information suitable for acceleration and enhancement of adequate sensorimotor representations and processible below the level of consciousness. In the experimental setting, participants were asked to learn a closed motor skill (technique acquisition of indoor rowing). One group was treated with visual information and two groups with audiovisual information (sonification vs. natural sounds). For all three groups learning became evident and remained stable. Participants treated with additional movement sonification showed better performance compared to both other groups. Results indicate that movement sonification enhances motor learning of a complex gross motor skill—even exceeding usually expected acoustic rhythmic effects on motor learning.

Acceleration and deceleration at constant speed: systematic modulation of motion perception by kinematic sonification

“…At about 70-80 ms, Mendonca et al 29 draw a closer temporal window for efficient audiovisual integration related to behavioral features compared to Stein and Meredith 10 related to single-neuron neurophysiology of primates. Young et al 32 also demonstrated that synthesized footstep sounds are perceptually processed like ecological footstep sounds on stride length, cadence estimations, and adaptation, supporting the concept of real-time kinematic movement sonification used by Effenberg 21 and Effenberg et al 22 Most of the referenced studies mapped the additional acoustics to distal segments or parts of the acting person (hand or hands, 3,39,40,42 feet, 29 pen-tip, 36,37 hands, and feet). 22 In addition, Vinken et al 17 drafted a mapping-concept explicitly referencing the "effectors' endpoint trajectory" (p. 537) and stated: "Movement sonification was used to transform kinematic data of the distal end effector into the acoustic domain" (p. 539).…”

“…Although Dyer et al . found transient effects of concurrent rhythmic sonification on a bimanual 4:3 shape‐tracing task resolving in a 24‐h retention measure, Effenberg et al . reported persistent effects of dynamic and kinematic real‐time sonification on motor learning of indoor rowing of novices—even beyond effects of rhythmic adjustments.…”

“…19 The used intermodal mapping and coding strategy was built on the basic natural relation between kinetic and acoustic event categories as described in the ecological approach to acoustic perception by Carello et al 20 and as already adapted to movement sonification. 21,22 A well-known example of a supramodal fundamental feature category is energy, which is defined within the auditory domain by the amplitude of a sound and within the kinetic domain by the kinetic energy and the potential energy. Even though movement kinematics are usually perceived visually (also designated as "biological motion perception," see Troje 23 ), selected kinematic parameters were transformed here into the auditory domain.…”

Training proprioception with sound: effects of real‐time auditory feedback on intermodal learning

“…[5][6][7] Moreover, such a feedback can enrich the develop-ment of perceptomotor representations by amplifying the brain's ability to integrate multiple congruent perceptual streams, therefore aiding in the formation of stable internal feed-forward models. 3,8 Research conclusively suggests that mapping a performer's action with real-time auditory feedback can enhance both the perceptuomotor representations in the brain and motor performance. [8][9][10][11][12] Strong influence of real-time auditory feedback on motor performance was thought to be due to its influence over the proprioceptive modality.…”

“…3,8 Research conclusively suggests that mapping a performer's action with real-time auditory feedback can enhance both the perceptuomotor representations in the brain and motor performance. [8][9][10][11][12] Strong influence of real-time auditory feedback on motor performance was thought to be due to its influence over the proprioceptive modality. [13][14][15][16] Hasegawa et al, 16 for instance, reported that training with auditory augmented biofeedback might facilitate the integration of auditory and proprioceptive systems.…”