Feeling the Effort of Classical Musicians - A Pipeline from Electromyography to Smartphone Vibration for Live Music Performance

Abstract

This paper presents the MappEMG pipeline. The goal of this pipeline is to augment the traditional classical concert experience by giving listeners access, through the sense of touch, to an intimate and non-visible dimension of the musicians' bodily experience while performing. The live-stream pipeline produces vibrations based on muscle activity captured through surface electromyography (EMG). Therefore, MappEMG allows the audience to experience the performer's muscle effort, an essential component of music performance which is typically unavailable to direct visual observation. The paper is divided in four sections. First, we overview related works on EMG, music performance, and vibrotactile feedback. We then present conceptual and methodological issues of capturing musicians' muscle effort related to their expressive intentions. We further explain the different components of the live-stream data pipeline: a python software named Biosiglive for data acquisition and processing, a Max/MSP patch for data post-processing and mapping, and a mobile application named hAPPtiks for real-time control of smartphones' vibration. Finally, we address the application of the pipeline in an actual music performance. Thanks to their modular structure, the tools presented could be used in different creative and biomedical contexts involving gestural control of haptic stimuli.

Multimodal Embodiment Research of Oral Music Traditions: Electromyography in Oud Performance and Education Research of Persian Art Music

“…For this reason, a strong turn towards the study of EMG data has been observed over the last few years in the music domain too [77]-the assumption behind this being a simple, unequivocal proportional relationship between EMG, underlying muscular forces, and effort, one of the presumptions that I wish to critically examine in this paper. In music research, EMG technologies have been used extensively, both as sensors to study expression in musicrelated gestures [57,76,[78][79][80][81][82][83][84] and as actuators offering vibrotactile feedback [85,86] to enhance instrument-learning practices [87] and provide additional multimodal feedback for those with auditory impairments [88]. For instance, a pilot study (reported in [89]) explored the relationship between effort-related EMG data and musical tension in Iannis Xenakis' piano composition 'Evryali', aiming to address expressed concerns by [90] regarding virtuosity, performability, physical exertion, and energy consumption, or 'energetic striving' [85], in the challenging passages of the work, notorious for the difficulty imposed by the dense and complex graphical notation.…”

“…This assumption is evident in various applications that infer muscular force from the monitored muscular activation levels through EMG data amplitude [137]. While this may be acceptable for non-medical applications in which sophisticated biomechanical analyses of high accuracy or methodological justifications are not critical, it contrasts with the selection of high-end medical-grade equipment in the context of musical practice when establishing conceptual links to muscular force, as seen in studies like [85], or occasionally by [138].…”

Biosiglive: an Open-Source Python Package for Real-time Biosignal Processing

“…biosiglive is divided into five independent modules. The main features are: Research Projects Using biosiglive (Verdugo et al, 2022)…”