Sonification of the self vs. sonification of the other: Differences in the sonification of performed vs. observed simple hand movements

Bresin, Roberto; Mancini, Maurizio; Elblaus, Ludvig; Frid, Emma

doi:10.1016/j.ijhcs.2020.102500

Cited by 8 publications

(6 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human locomotion may influence, and interact with, emotions, and locomotion sounds may be influenced by the emotion of the walker (Giordano & Bresin, 2006). Sounds produced on a firm surface lead to more aggressive walking patterns (Bresin, de Witt, Papetti, Civolani, & Fontana, 2010). Runners were shown to alter step length and, thereby, speed when presented with music of different emotional character ("relaxing" or "activating"), while retaining a pace of 130 beats per minute (BPM) (Leman et al, 2013).…”

Section: Human Walking Soundmentioning

confidence: 99%

Bipedal Steps in the Development of Rhythmic Behavior in Humans

2019

View full text Add to dashboard Cite

We contrast two related hypotheses of the evolution of dance: H1: Maternal bipedal walking influenced the fetal experience of sound and associated movement patterns; H2: The human transition to bipedal gait produced more isochronous/predictable locomotion sound resulting in early music-like behavior associated with the acoustic advantages conferred by moving bipedally in pace. The cadence of walking is around 120 beats per minute, similar to the tempo of dance and music. Human walking displays long-term constancies. Dyads often subconsciously synchronize steps. The major amplitude component of the step is a distinctly produced beat. Human locomotion influences, and interacts with, emotions, and passive listening to music activates brain motor areas. Across dance-genres the footwork is most often performed in time to the musical beat. Brain development is largely shaped by early sensory experience, with hearing developed from week 18 of gestation. Newborns reacts to sounds, melodies, and rhythmic poems to which they have been exposed in utero. If the sound and vibrations produced by footfalls of a walking mother are transmitted to the fetus in coordination with the cadence of the motion, a connection between isochronous sound and rhythmical movement may be developed. Rhythmical sounds of the human mother locomotion differ substantially from that of nonhuman primates, while the maternal heartbeat heard is likely to have a similar isochronous character across primates, suggesting a relatively more influential role of footfall in the development of rhythmic/musical abilities in humans. Associations of gait, music, and dance are numerous. The apparent absence of musical and rhythmic abilities in nonhuman primates, which display little bipedal locomotion, corroborates that bipedal gait may be linked to the development of rhythmic abilities in humans. Bipedal stimuli in utero may primarily boost the ontogenetic development. The acoustical advantage hypothesis proposes a mechanism in the phylogenetic development.

show abstract

Section: Human Walking Soundmentioning

confidence: 99%

Bipedal Steps in the Development of Rhythmic Behavior in Humans

2019

View full text Add to dashboard Cite

show abstract

“…A key distinction between our movement materials and real-life pathological STS pattern categories is that the latter are likely to be less clear-cut , manifesting as combinations of discrete impairments resulting from muscle weakness and/or spasticity (Cheng et al, 1998 ; Scarborough et al, 2007 ; Boukadida et al, 2015 ). The sonification design strongly depends on the use-case scenario within the context of feedback and monitoring (Hildebrandt et al, 2014 ; Millor et al, 2014 ; Bresin et al, 2020 ). For example, a sonification of jerk may be useful for a therapist who is monitoring/assessing a patient, but not for feedback purposes, as patients may not be able to spontaneously convert this information into meaningful biomechanical change.…”

Section: Discussionmentioning

confidence: 99%

An Embodied Sonification Model for Sit-to-Stand Transfers

2022

View full text Add to dashboard Cite

Interactive sonification of biomechanical quantities is gaining relevance as a motor learning aid in movement rehabilitation, as well as a monitoring tool. However, existing gaps in sonification research (issues related to meaning, aesthetics, and clinical effects) have prevented its widespread recognition and adoption in such applications. The incorporation of embodied principles and musical structures in sonification design has gradually become popular, particularly in applications related to human movement. In this study, we propose a general sonification model for the sit-to-stand (STS) transfer, an important activity of daily living. The model contains a fixed component independent of the use-case, which represents the rising motion of the body as an ascending melody using the physical model of a flute. In addition, a flexible component concurrently sonifies STS features of clinical interest in a particular rehabilitative/monitoring situation. Here, we chose to represent shank angular jerk and movement stoppages (freezes), through perceptually salient pitch modulations and bell sounds. We outline the details of our technical implementation of the model. We evaluated the model by means of a listening test experiment with 25 healthy participants, who were asked to identify six normal and simulated impaired STS patterns from sonified versions containing various combinations of the constituent mappings of the model. Overall, we found that the participants were able to classify the patterns accurately (86.67 ± 14.69% correct responses with the full model, 71.56% overall), confidently (64.95 ± 16.52% self-reported rating), and in a timely manner (response time: 4.28 ± 1.52 s). The amount of sonified kinematic information significantly impacted classification accuracy. The six STS patterns were also classified with significantly different accuracy depending on their kinematic characteristics. Learning effects were seen in the form of increased accuracy and confidence with repeated exposure to the sound sequences. We found no significant accuracy differences based on the participants' level of music training. Overall, we see our model as a concrete conceptual and technical starting point for STS sonification design catering to rehabilitative and clinical monitoring applications.

show abstract

“…Since each camera performed a separate head tracking, they had to be merged before using them. The processing was carried out via a Python script using the OpenCV library [15]. We applied linear interpolation and average smoothing in case of missing frames.…”

Section: Head Position Featuresmentioning

confidence: 99%

Modelling the “transactive memory system” in multimodal multiparty interactions

Biancardi,

Mancini,

Ravenet

et al. 2023

J Multimodal User Interfaces

Self Cite

View full text Add to dashboard Cite

Transactive memory system (TMS) is a team emergent state representing the knowledge of each member about “who knows what” in a team performing a joint task. We present a study to show how the three TMS dimensions Credibility, Specialisation, Coordination, can be modelled as a linear combination of the nonverbal multimodal features displayed by the team performing the joint task. Results indicate that, to some extent, the three dimensions of TMS can be expressed as a linear combination of nonverbal multimodal features. Moreover, the higher the number of modalities (audio, movement, spatial), the better the modelling. Results could be used in future work to design human-centered computing applications able to automatically estimate TMS from teams’ behavioural patterns, to provide feedback and help teams’ interactions.

show abstract

Sonification of the self vs. sonification of the other: Differences in the sonification of performed vs. observed simple hand movements

Cited by 8 publications

References 8 publications

Bipedal Steps in the Development of Rhythmic Behavior in Humans

Bipedal Steps in the Development of Rhythmic Behavior in Humans

An Embodied Sonification Model for Sit-to-Stand Transfers

Modelling the “transactive memory system” in multimodal multiparty interactions

Contact Info

Product

Resources

About