Motor Variability in Complex Gesture Learning: Effects of Movement Sonification and Musical Background

Abstract

With the increasing interest in movement sonification and expressive gesture-based interaction, it is important to understand which factors contribute to movement learning and how. We explore the effects of movement sonification and users’ musical background on motor variability in complex gesture learning. We contribute an empirical study in which musicians and non-musicians learn two gesture sequences over three days, with and without movement sonification. Results show the interlaced interaction effects of these factors and how they unfold in the three-day learning process. For gesture 1, which is fast and dynamic with a direct “action-sound” sonification, movement sonification induces higher variability for both musicians and non-musicians on day 1. While musicians reduce this variability to a similar level as no auditory feedback condition on day 2 and day 3, non-musicians remain to have significantly higher variability. Across three days, musicians also have significantly lower variability than non-musicians. For gesture 2, which is slow and smooth with an “action-music” metaphor, there are virtually no effects. Based on these findings, we recommend future studies to take into account participants’ musical background, consider longitudinal study to examine these effects on complex gestures, and use awareness when interpreting the results given a specific design of gesture and sound.

Describing movement learning using metric learning

“…We used a publicly available dataset, previously used in a motor learning study [21] where it was investigated how participants learned from video a complex hand movement, referred to The template gesture was performed by the person who designed the gesture, seated on a chair, wearing on the right hand a custom-made glove equipped with optical markers and an IMU (Inertial Measurement Units), while being filmed (Fig 2, top panel). The gesture was specifically designed to be performed with the hand, so most of the information was indeed carried by the hand.…”

“…The video was captured at 30fps. For more details on the experimental protocol used in the dataset collection, please refer to [21].…”

“…The learning rates are reported below the optimisation weights in Fig 7 . We found a significant linear relationships between the means of optimised weights and learning rates (r(25) = −0.70, p < 0.001). By comparing adjacent segments with t-tests, we identified three main regions of motor improvement located in segments [7][8][9], [12][13][14] and [16][17][18][19][20][21]. The beginning and end of the movement showed the smallest learning rates.…”

Effects of pitch and musical sounds on body-representations when moving with sound

“…This illusion was replicated both in adults and pre-school children for passive finger pulling 37 Influence of harmonic stability of musical sounds on perceived body position: Musically resolved or unresolved sonifications accompanying squat movements impact on the perceived depth of the squat 38 Influence of movement sonification on movement variability: Movement sonification can induce higher movement variability for both musicians and non-musicians when starting to learn a new movement sequence, while it is reduced later when the movement is mastered. It has been hypothesized that the sound feedback provokes a change of attentional focus that perturbates proprioceptive awareness 39 Bodily and emotional feelings Feelings about one's body, the movement, and emotional state: A sound increasing or decreasing in pitch accompanying the participant's arm movement will impact on how people feel "about their body" (e.g., weight or speed) and "about their movement" (e.g., ease, comfort). Sounds increasing vs. decreasing in pitch will enhance the emotional state, making people feel happier, more excited and motivated…”

Effect of sonification types in upper-limb movement: a quantitative and qualitative study in hemiparetic and healthy participants

“…Although the intrinsic principle of sonification devices is conducive to external focus and implicit learning (Dyer, 2015), the attentional processes mobilized during the use of sonification devices remain insufficiently 36 known to date. In particular, it would be necessary to study the strategies used according to the users' experience in order to determine in which cases sound feedback can be considered as distractors (Parmentier, 2014Liu, 2022, sources of external focus, or even sources of internal focus if we consider that an optimized mapping could be likely to favor attention to proprioception. In the case of rehabilitation, it is commonly accepted that it is important to limit attentional distractors and that dual-task situations can be too costly and diminish motor performance in the case of gait (Montero-Odasso, 2012).…”