Auditory Coding of Human Movement Kinematics

Abstract

Although visual perception is dominant on motor perception, control and learning, auditory information can enhance and modulate perceptual as well as motor processes in a multifaceted manner. During last decades new methods of auditory augmentation had been developed with movement sonification as one of the most recent approaches expanding auditory movement information also to usually mute phases of movement. Despite general evidence on the effectiveness of movement sonification in different fields of applied research there is nearly no empirical proof on how sonification of gross motor human movement should be configured to achieve information rich sound sequences. Such lack of empirical proof is given for (a) the selection of suitable movement features as well as for (b) effective kinetic-acoustical mapping patterns and for (c) the number of regarded dimensions of sonification. In this study we explore the informational content of artificial acoustical kinematics in terms of a kinematic movement sonification using an intermodal discrimination paradigm. In a repeated measure design we analysed discrimination rates of six everyday upper limb actions to evaluate the effectiveness of seven different kinds of kinematic-acoustical mappings as well as short-term learning effects. The kinematics of the upper limb actions were calculated based on inertial motion sensor data and transformed into seven different sonifications. Sound sequences were randomly presented to participants and discrimination rates as well as confidence of choice were analysed. Data indicate an instantaneous comprehensibility of the artificial movement acoustics as well as short-term learning effects. No differences between different dimensional encodings became evident thus indicating a high efficiency for intermodal pattern discrimination for the acoustically coded velocity distribution of the actions. Taken together movement information related to continuous kinematic parameters can be transformed into the auditory domain. Additionally, pattern based action discrimination is obviously not restricted to the visual modality. Artificial acoustical kinematics might be used to supplement and/or substitute visual motion perception in sports and motor rehabilitation.

Acceleration and deceleration at constant speed: systematic modulation of motion perception by kinematic sonification

“…Young et al 32 also demonstrated that synthesized footstep sounds are perceptually processed like ecological footstep sounds on stride length, cadence estimations, and adaptation, supporting the concept of real-time kinematic movement sonification used by Effenberg 21 and Effenberg et al 22 Most of the referenced studies mapped the additional acoustics to distal segments or parts of the acting person (hand or hands, 3,39,40,42 feet, 29 pen-tip, 36,37 hands, and feet). 22 In addition, Vinken et al 17 drafted a mapping-concept explicitly referencing the "effectors' endpoint trajectory" (p. 537) and stated: "Movement sonification was used to transform kinematic data of the distal end effector into the acoustic domain" (p. 539).…”

“…The core idea is an auditory coding of movement kinematics, which has already been introduced and investigated by our workgroup . Research on the inherent information of natural movement‐attendant sounds indicates a rich spectrum of different kinds of information, such as for agent identification and discrimination with complex natural movement sounds or even related to temporal deviations in tap dance sequences .…”

“…The core idea is an auditory coding of movement kinematics, which has already been introduced and investigated by our workgroup. 17 Research on the inherent information of natural movementa For the stimuli used, described in detail within the section "Stimulus material," the term "relative movement velocity" can be understood also as movement frequency. Though the center of the pelvis is used as the origin of the coordinate system, only relative movements can be observed in order to estimate the velocity of the movement, absolute (i.e., translational) movement of the swimmer does not take place.…”

Transposing musical skill: sonification of movement as concurrent augmented feedback enhances learning in a bimanual task

“…Behavioural effects are especially strong for extensively practiced noisy actions, for example instrumental performance (Taylor & Witt, 2015). Additionally, specific actions can even be identified from their sonified velocity profile alone (Vinken et al, 2013). Summarised, sound and movement are ecologically coupled.…”

Audio cues enhance mirroring of arm motion when visual cues are scarce

“…Pitch-based auditory cues provide an informative, intuitive and commonly used approach for representing kinematic and kinetic measurements in human motion [10][11][12]. When used as feedback, audio cues can enhance performance at motor tasks across a variety of contexts including learning a cyclic motion [10,13,14] and interpersonal coordination [2,15,16]-more generally perceptual coupling including other sensory modalities like vision and touch have been shown to enhance interpersonal coordination (see [17] for a review).…”