From expressive gesture to sound

Abstract

This paper contributes to the development of a multimodal, musical tool that extends the natural action range of the human body to communicate expressiveness into the virtual music domain. The core of this musical tool consists of a low cost, highly functional computational model developed upon the Max/MSP platform that (1) captures real-time movement of the human body into a 3D coordinate system on the basis of the orientation output of any type of inertial sensor system that is OSC-compatible, (2) extract low-level movement features that specify the amount of contraction/expansion as a measure of how a subject uses the surrounding space, (3) recognizes these movement features as being expressive gestures, and (4) creates a mapping trajectory between these expressive gestures and the sound synthesis process of adding harmonic related voices on an in origin monophonic voice. The concern for a user-oriented and intuitive mapping strategy was thereby of central importance. This was achieved by conducting an empirical experiment based on theoretical concepts from the embodied music cognition paradigm. Based on empirical evidence, this paper proposes a mapping trajectory that fa-P.-J. Maes ( ) · M. Leman · M. Lesaffre · M. Demey · cilitates the interaction between a musician and his instrument, the artistic collaboration between (multimedia) artists and the communication of expressiveness in a social, musical context.

Action-based effects on music perception

“…Research has shown that these body movements often reflect the performer's movements from which the music originated (Leman et al, 2009; Godøy and Leman, 2010), certain aspects of the melody, harmony, rhythm and timbre (Maes et al, 2010; Naveda and Leman, 2010; Toiviainen et al, 2010; Burger et al, 2013; Leman et al, 2013), or even the listeners' mood (Van Dyck et al, 2013). These and similar studies importantly indicate that the listeners' musical mind (attention, intention, mood, feelings, etc.)…”

Choreography in the Mapping of New Instruments

“…'The basic problem of these interfaces concerns the fact that the mediation between the different modalities (basically from movement to sound) has an arbitrary component, which is due to the fact that the energies of the modalities are transformed into electronic signals. This is in contrast with traditional instruments, where energetic modalities are mechanically mediated and where the user gets a natural feeling of the causality of the multimodal interface' [1].…”

“…These three factors cannot coexist and hence must be weighed up against each other. Using these three factors, we outline our approach to the 'mapping problem' [1] in designing a new gestural instrument for electronic percussion. We use the analogy of a dancer choreographing her/his movement to prerecorded sound in taking a compositional approach to mapping, beginning with the music and working backwards to the choreography of the movement on the gestural instrument to best fit with the music.…”

A Natural User Interface for Gestural Expression and Emotional Elicitation to Access the Musical Intangible Cultural Heritage

“…Leveraging the above, there is a growing interest in the analysis of the gestural knowledge. A large amount of studies conducted in the last years on embodied music cognition have investigate that not only effective, but also accompanist and figurative gestures are very important, since they are related to the expressivity and to the social interaction of the performer with the audience [Jensenius 2007;Maes et al 2010]. However, "learning" musical gestures and "performing" music are usually perceived as separate concepts and experiences that pass through intermediate physical mechanisms.…”