Combining EEG, MIDI, and motion capture techniques for investigating musical performance

Abstract

This article describes a setup for the simultaneous recording of electrophysiological data (EEG), musical data (MIDI), and three-dimensional movement data. Previously, each of these three different kinds of measurements, conducted sequentially, has been proven to provide important information about different aspects of music performance as an example of a demanding multisensory motor skill. With the method described here, it is possible to record brain-related activity and movement data simultaneously, with accurate timing resolution and at relatively low costs. EEG and MIDI data were synchronized with a modified version of the FTAP software, sending synchronization signals to the EEG recording device simultaneously with keypress events. Similarly, a motion capture system sent synchronization signals simultaneously with each recorded frame. The setup can be used for studies investigating cognitive and motor processes during music performance and music-like tasks--for example, in the domains of motor control, learning, music therapy, or musical emotions. Thus, this setup offers a promising possibility of a more behaviorally driven analysis of brain activity.

Predictive error detection in pianists: a combined ERP and motion capture study

“…The present study investigated errors during the performance of musical sequences by using a novel setup combining the recording of EEG, MIDI, and 3D movement data (for details of this setup, see Maidhof et al, 2013; for a non-musical setup aiming at mobile EEG recording of freely moving participants, see Makeig et al, 2009). This allowed us to investigate the underlying movements of pitch errors during piano performances, but also to exploratively relate the neurophysiological findings to different movement phases.…”

“…By contrast, when the auditory feedback of keystrokes or of speech acts is externally manipulated, the earliest information with which the predicted consequences can be compared is the auditory feedback. Thus, brain responses after the onset of the auditory feedback are increased (or inhibitory effects are canceled) when a mismatch is detected (e.g., Houde et al, 2002; Heinks-Maldonado et al, 2006; Eliades and Wang, 2008; Katahira et al, 2008; Maidhof et al, 2013, 2010; Behroozmand and Larson, 2011; Behroozmand et al, 2011). However, when participants commit errors, information available even earlier, including tactile feedback, could be used for error detection.…”

“…To synchronize MIDI, motion capture, and EEG data, the FTAP software sent synchronization signals concurrently with every fifth key press to the EEG recording device. Similarly, the motion capture system sent synchronization signals simultaneously with each recorded frame to the EEG recording device (for details of this setup, see Maidhof et al, 2013). …”

Methodological aspects of EEG and body dynamics measurements during motion

“…This method requires a master timebase, which regularly sends out synchronization pulses to the attached devices. Nowadays camera systems such as Qualisys Oqus system provide either external frequency outputs or locking into external synchronization input pulse sequences (Maidhof et al, 2013). …”

Measuring Aksak Rhythm and Synchronization in Transylvanian Village Music by Using Motion Capture

“…The tracking of two markers only, positioned at the top of the violin and viola bows, enabled us to analyze the S:L proportion of the aksak pattern and the different types and degrees of asynchrony with a higher precision than the measures obtained from the sound signal. While it is widely accepted that 'embodied' representations of the musical activity may offer new possibilities for a deeper understanding of music production and perception (among others, see Leman, 2008;Danielsen, 2010;Burger, 2013;Maidhof et al, 2014), we believe that they may also serve as a support for music analysis and transcription, thus contributing to an emerging nonethnocentric empirical musicology.…”