If motion sounds: Movement sonification based on inertial sensor data

Brock, Heike; Schmitz, Gerd; Baumann, Jan; Effenberg, Alfred O.

doi:10.1016/j.proeng.2012.04.095

Cited by 18 publications

(10 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data of the inertial sensors were recorded while the model was executing reaching actions with the digitizing pen towards one of nine different target positions on the tablet (see Figure 3). In a next step, a forward kinematic model of the right arm was used to determine trajectory and velocity information of the right metacarpophalangeal joints [54]. Four specific kinematic parameters were chosen to be mapped on four different parameters of a sound (patch 100 "jupiter lead" from "Preset D Group", SonicCell, Roland) [54]: (a) The absolute velocity of the right metacarpophalangeal joints was mapped on the volume of the sound (the faster the movement, the louder the sound).…”

Section: Stimulus Materialsmentioning

confidence: 99%

“…In a next step, a forward kinematic model of the right arm was used to determine trajectory and velocity information of the right metacarpophalangeal joints [54]. Four specific kinematic parameters were chosen to be mapped on four different parameters of a sound (patch 100 "jupiter lead" from "Preset D Group", SonicCell, Roland) [54]: (a) The absolute velocity of the right metacarpophalangeal joints was mapped on the volume of the sound (the faster the movement, the louder the sound). (b) The position of the right metacarpophalangeal joints on the y-axis was mapped on the pitch (the higher the metacarpophalangeal joints in space, the higher the sound).…”

Section: Stimulus Materialsmentioning

confidence: 99%

See 1 more Smart Citation

Auditory Coding of Reaching Space

et al. 2020

Self Cite

View full text Add to dashboard Cite

Reaching movements are usually initiated by visual events and controlled visually and kinesthetically. Lately, studies have focused on the possible benefit of auditory information for localization tasks, and also for movement control. This explorative study aimed to investigate if it is possible to code reaching space purely by auditory information. Therefore, the precision of reaching movements to merely acoustically coded target positions was analyzed. We studied the efficacy of acoustically effect-based and of additional acoustically performance-based instruction and feedback and the role of visual movement control. Twenty-four participants executed reaching movements to merely acoustically presented, invisible target positions in three mutually perpendicular planes in front of them. Effector-endpoint trajectories were tracked using inertial sensors. Kinematic data regarding the three spatial dimensions and the movement velocity were sonified. Thus, acoustic instruction and real-time feedback of the movement trajectories and the target position of the hand were provided. The subjects were able to align their reaching movements to the merely acoustically instructed targets. Reaching space can be coded merely acoustically, additional visual movement control does not enhance reaching performance. On the basis of these results, a remarkable benefit of kinematic movement acoustics for the neuromotor rehabilitation of everyday motor skills can be assumed.

show abstract

Section: Stimulus Materialsmentioning

confidence: 99%

Section: Stimulus Materialsmentioning

confidence: 99%

Auditory Coding of Reaching Space

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…The videotaped sequences were used for later visual instruction (see Section 2.3). In a next step, a four-segment forward kinematic model of the right arm was used to determine position and velocity information of the right hand needed for the sonification (Brock et al, 2012).…”

Section: Preparation Of Sonificationsmentioning

confidence: 99%

“…Four specific kinematic parameters from the four-segment model were selected to modulate four different sound parameters based on the sound 'Jupiter Lead', which was generated by the Synth Modul 'SonicCell' (Roland Germany GmbH, Nauheim, Germany; Brock et al, 2012). Key features of sounds are their frequency, amplitude or intensity, spectral patterns and -in the case of an azimuth angle different from zero -temporal and intensity differences between ears.…”

Section: Preparation Of Sonificationsmentioning

confidence: 99%

Auditory Coding of Human Movement Kinematics

et al. 2013

Self Cite

View full text Add to dashboard Cite

Although visual perception is dominant on motor perception, control and learning, auditory information can enhance and modulate perceptual as well as motor processes in a multifaceted manner. During last decades new methods of auditory augmentation had been developed with movement sonification as one of the most recent approaches expanding auditory movement information also to usually mute phases of movement. Despite general evidence on the effectiveness of movement sonification in different fields of applied research there is nearly no empirical proof on how sonification of gross motor human movement should be configured to achieve information rich sound sequences. Such lack of empirical proof is given for (a) the selection of suitable movement features as well as for (b) effective kinetic-acoustical mapping patterns and for (c) the number of regarded dimensions of sonification. In this study we explore the informational content of artificial acoustical kinematics in terms of a kinematic movement sonification using an intermodal discrimination paradigm. In a repeated measure design we analysed discrimination rates of six everyday upper limb actions to evaluate the effectiveness of seven different kinds of kinematic-acoustical mappings as well as short-term learning effects. The kinematics of the upper limb actions were calculated based on inertial motion sensor data and transformed into seven different sonifications. Sound sequences were randomly presented to participants and discrimination rates as well as confidence of choice were analysed. Data indicate an instantaneous comprehensibility of the artificial movement acoustics as well as short-term learning effects. No differences between different dimensional encodings became evident thus indicating a high efficiency for intermodal pattern discrimination for the acoustically coded velocity distribution of the actions. Taken together movement information related to continuous kinematic parameters can be transformed into the auditory domain. Additionally, pattern based action discrimination is obviously not restricted to the visual modality. Artificial acoustical kinematics might be used to supplement and/or substitute visual motion perception in sports and motor rehabilitation.

show abstract

“…Another reason is the rising number of applications. For instance, sonification of movements [8,10], analysis of sports-, rehabilitation-, and healthcare sessions [14,21,22,32] as well as applications within the clinical and veterinary field [18,24]. These applications require an accurate and precise recognition of actions and movements leading to a challenging topic in machine learning.…”

Section: Introductionmentioning

confidence: 99%

Multi-sensor Acceleration-Based Action Recognition

Baumann

Schulz

Rosenhahn

2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. In this paper, a framework to recognize human actions from acceleration data is proposed. An important step for an accurate recognition is the preprocessing of input data and the following classification by the machine learning algorithm. In this paper, we suggest to combine Dynamic Time Warping (DTW) with Random Forest. The intention of using DTW is to pre-process the data to eliminate outliers and to align the time series. Many applications require more than one inertial sensor for an accurate prediction of actions. In this paper, nine inertial sensors are deployed to ensure an accurate recognition of actions. Further, sensor fusion approaches are introduced and the most promising strategy is shown. The proposed framework is evaluated on a self-recorded dataset consisting of six human actions. Each action was performed three times by 20 subjects. The dataset is publicly available for download.

show abstract

If motion sounds: Movement sonification based on inertial sensor data

Cited by 18 publications

References 6 publications

Auditory Coding of Reaching Space

Auditory Coding of Reaching Space

Auditory Coding of Human Movement Kinematics

Multi-sensor Acceleration-Based Action Recognition

Contact Info

Product

Resources

About