Understanding musical expressiveness using interactive multimedia platforms

Abstract

Background in musicology Understanding the gesture-based foundations of musical involvement opens a number of new perspectives for musicology, with the likely effect of a change in approach. Giving justice to the role of gesture in music calls for an integrated view of perception and action, which implies a shift from auditory-based cognition towards embodied cognition, an approach that includes the whole human body as mediator between mental processes and physical energy. Background in computing With the help of new measurement tools, new computational models of multi-modal interaction, and new interactive multimedia platforms, it becomes easier to study music from a multi-modal point of view, both in terms of scientific and artistic perspectives. Music also offers an appropriate domain for the testing and developing of interactive systems and computational models for handling non-verbal multi-modal communication. Aims In studying musical gesture, both musicology and engineering can profit from a joint collaboration. While technology provides new tools for the measurement and modelling of musical involvement, knowledge of its corporeal foundation may largely contribute to the development of technology that in turn contributes to measurement and modelling. Furthermore, this knowledge may be exploited in artistic applications. As such, the interaction between musicology and music engineering may lead to interesting developments in the domain of embodied cognition and interactive multimedia platforms. Main contribution An overview of recent work on the development of an interactive multimedia platform for the measurement and modelling of musical expressiveness is presented. The paper introduces the notion of corporeal imitation and mirroring behavior as a core concept fur understanding musical expressiveness. Implications The approach advocated in this paper implies a shift of paradigm in systematic musicology, away from an all too narrow focus on auditory perception and sound structure, towards a broader view that encompasses multi-modal musical involvement. This approach is grounded in multi-sensory integration, the coupling of perception and action, the study of motor imitation, and issues that relate to affect, emotions and subjectivity. The advantage of using a interactive multimedia platform for music research, rather than ad hoc solutions, pertains to the use of modular functionalities, which allow rapid prototyping of experimental setup, flexible modelling, and the subsequent use of results in artistic applications.

Analyzing Expressive Qualities in Movement and Stillness: Effort-Shape Analyses of Solo Marimbists' Bodily Expression

“…Mirroring observed bodily expressionas stipulated by the effort-shape systemmay have contributed to the high interjudge reliability observed in temporal positioning and frequency of observation for nongoaldirected (nonmarimba-playing) expressive transformation drive and shape movements. The mirroring process, whether covert or actual, may play a role in understanding the expressive intentions of others (Bartenieff & Lewis, 1980;Cox, 2001;Laban, 1988;Leman & Camurri, 2006;Moore & Yamamoto, 1988). The hypothesized mechanism at work is the human mirror neuron system transforming audio-visual sensory information into motor activation in participants, in turn, associated with understanding the performers' intentions (Molnar-Szakacs & Overy, 2006).…”

“…Many authors have proposed that covert or overt imitation of observed movements of others may play a role in perceiving and understanding the intention underlying such actions (Cox, 2001;Leman & Camurri, 2006;Moore & Yamamoto, 1988). Simulation theory posits that others' mental states are understood through adopting a similar mental state, or by -putting oneself in another's shoes‖ (Gallese & Goldman, 1998).…”

From expressive gesture to sound

“…Such a solution would be based on experiments that probe the natural tendencies for multimodal mappings. The theoretical grounding for such approach is found in the embodied music cognition paradigm, in which the importance of gesture and corporeal imitation as a basis of understanding musical expressiveness is stressed [26,27]. In this view, multimodal interfaces for music are approached in terms of mediation technologies that connect with actionrelevant cues (i.e.…”

The “One-Person Choir”: A Multidisciplinary Approach to the Development of an Embodied Human–Computer Interface

“…This approach is grounded in multisensory integration, the coupling of perception and action, the study of motor imitation, and issues that relate to affect, emotions, and subjectivity (Camurri et al 2005;Leman and Camurri 2006;Maes et al 2010).…”