The Sense of Ensemble: a Machine Learning Approach to Expressive Performance Modelling in String Quartets

Abstract

Computational approaches for modeling expressive music performance have produced systems that emulate music expression, but few steps have been taken in the domain of ensemble performance. In this paper, we propose a novel method for building computational models of ensemble expressive performance and show how this method can be applied for deriving new insights about collaboration among musicians. In order to address the problem of interdependence among musicians we propose the introduction of inter-voice contextual attributes. We evaluate the method on data extracted from multi-modal recordings of string quartet performances in two different conditions: solo and ensemble. We used machine-learning algorithms to produce computational models for predicting intensity, timing deviations, vibrato extent, and bowing speed of each note. As a result, the introduced inter-voice contextual attributes generally improved the prediction of the expressive parameters. Furthermore, results on attribute selection show that the models trained on ensemble recordings took more advantage of inter-voice contextual attributes than those trained on solo recordings.

Computational Models of Expressive Music Performance: A Comprehensive and Critical Review

“…Caramiaux et al (2017) study performers' skill levels through variability in timing and features describing finger motion. Marchini et al (2014) study the use of score features describing horizontal (i.e., melodic) and vertical (i.e., harmonic) contexts for modeling dynamics, articulation, and timbral characteristics of expressive ensemble performances, focusing on string quartets. Using machine learning and feature selection techniques, Giraldo S.I.…”

Enriched Multimodal Representations of Music Performances: Online Access and Visualization

“…Our empirical study of multimodal string quartet performance data has been motivated by three main objectives: studying the interdependence between the members of a string quartet ensemble, training models of joint expressivity in ensemble music performance, and compiling/sharing research datasets that serve as both ground truth as well as a starting point for further research on music ensemble performance [10,9,12,13]. Either in accomplishing our own research goals, making the data available to other researchers, or in developing flexible platform for open multimodal data exchange, a number of technical and logistical challenges have been faced in this project.…”

Motion informed audio source separation

“…Thus, the motion information exists in the form of tracking data (motion capture or MoCap data ) acquired by sensors placed on each instrument and the bow [24]. Now we immediately recognize that information about "where" and "how" strongly the sound-producing object is excited will be readily conveyed by bowing motion velocity and orientation in time.…”

“…We use the publicly available Ensemble Expressive Performance (EEP) dataset 1 [26]. This dataset contains 23 multimodal recordings of string quartet performances (including both ensemble and solo).…”

“…In this light, we choose to use bow inclination (in degrees) and bow velocity (cm/s) as features (as shown in Fig. 1), which can be easily computed from the raw motion capture data described in [24,25]. These descriptors have been pre-computed and provided with the dataset.…”