Designing Action–Sound Metaphors Using Motion Sensing and Descriptor-Based Synthesis of Recorded Sound Materials

Abstract

In this chapter, we present a general methodology for designing interactive musical systems, using movement sensing and descriptor-based synthesis of recorded sound materials. Importantly, the design principles focus on action-sound metaphors that can be built upon audio features of recorded sound materials and their possible relationships to human movement. We describe the critical design choices, including sensing technologies, movement analysis, and action-sound models, along with application examples.

Motor Variability in Complex Gesture Learning: Effects of Movement Sonification and Musical Background

“…We chose a sonification approach based on different action-sound metaphors that are generally understandable by a wide audience [8,13]. Note that the goal of our study is to explore how sonification modulates motor variability in movement learning, rather than helps the learning process.…”

“…We used two different approaches to associate these two gestures with sounds in order to further differentiate how they will be perceived. The sonification of gesture 1 is based on an approach called "action-sound" metaphor that has been described in [8]. This takes advantage of relationships we acquire in our everyday interaction with objects.…”

“…In movement-based music systems (sometimes also referred to as embodied music systems) [8,80], gestures and movements are generally considered to convey various information layers simultaneously, allowing for example to trigger specific sounds, to modulate them and to communicate the performer's intention [38]. This requires performer to practice movement to reach such a conscious control of their action, thus implying sensorimotor learning [43].…”

Bodies moving with sound: Effects of pitch and musical sounds on body-representations

“…Systems that provide sound feedback on movement in real-time have been found to increase bodily awareness and influence movement (e.g., [48,49]) and are increasingly being used in the context of musical expression [89], dance (e.g., [90]), sports (e.g., [91][92][93]), general physical activity (e.g., [40,47]) and physical rehabilitation (e.g., [94,95]) for example, in people with chronic stroke [96][97][98][99], vestibular disorders (e.g., [100,101]), chronic pain (e.g., [49,51]) or autism (e.g., [102]). By contrast with those movement sonification scenarios, our studies do not rely on continuous real-time adjustment of the sound once it has been triggered.…”

Designing a gesture-sound wearable system to motivate physical activity by altering body perception

“…The sounds are produced using different sound synthesis engines and mapping strategies. First, we implemented a series of sound engines using descriptor-based concatenative synthesis [1]. This allows for playing small pieces of recorded sounds with a quasi-continuous control on their technique, implemented in Max (Cycling'74) with the library MuBu for Max (freely available):  Mappings 1-2: the FSRs in the shoes are mapped to either "water splash" sounds or to "aluminium can crash" sounds with various pitches, intensities, etc (related to paradigms 1, 2, 5, 7 in Table 2).…”