Classifying Biometric Data for Musical Interaction Within Virtual Reality

Abstract

Since 2015, commercial gestural interfaces have widened accessibility for researchers and artists to use novel Electromyographic (EMG) biometric data. EMG data measures musclar amplitude and allows us to enhance Human-Computer Interaction (HCI) through providing natural gestural interaction with digital media. Virtual Reality (VR) is an immersive technology capable of simulating the real world and abstractions of it. However, current commercial VR technology is not equipped to process and use biometric information. Using biometrics within VR allows for better gestural detailing and use of complex custom gestures, such as those found within instrumental music performance, compared to using optical sensors for gesture recognition in current commercial VR equipment. However, EMG data is complex and machine learning must be used to employ it. This study uses a Myo armband to classify four custom gestures in Wekinator and observe their prediction accuracies and representations (including or omitting signal onset) to compose music within VR. Results show that specific regression and classification models, according to gesture representation type, are the most accurate when classifying four music gestures for advanced music HCI in VR. We apply and record our results, showing that EMG biometrics are promising for future interactive music composition systems in VR.

Towards Developing a Virtual Guitar Instructor through Biometrics Informed Human-Computer Interaction

“…We also intend to build upon previous research in this area from the first author (see Figure 6) with this research enquiry, exploring how 3D virtual elements can be used within game engines (VR/MR) to provide intuitive feedback to guitar students. We will also be building upon our research in [27] to explore how the method presented in this work can be used with the game engine mechanics we built in that enquiry to create interactive music and visuals from EMG data (which is not native to current commercial VR equipment, despite being musically beneficial regarding HCI). This will allow us to decide which type of medium the instructor should be used within and the creative affordances of such medium.…”

“…In [9], performance technique is investigated on the violin instrument using orientation data (from Myo armbands) within set conditions, such as simple musical phrases and set windows of time, in order to observe, structure and understand the data. Our previous work [25][26][27] investigated how musical gestures for the piano (and gestures not using instruments ('in the air')) can be classified using biometric data, from the Myo armbands, with varying levels of efficacy, using continuous…”

“…Feature extraction must be used with EMG data in order to extract the most important and prominent features of the data, depending on the use case. Machine learning (ML) is also useful in this domain because it allows us to pair complex EMG information (representing musical gesture and performance) with interactive systems, such as game engines (as investigated during our earlier research [7,8,27]). Game engines offer unique opportunities for musical expression due to their developed physics engines and interactive/immersive features.…”

“…Within the technology industry, Apple and Facebook were recently granted patents for using biometrics within wearable technologies towards individualised feedback outside of music [23,24]. In this work, we build on our previous research which classified musical gestures using EMG data from Myo armbands and compared the classification efficacy/behaviour of several models using Wekinator [25][26][27] by adopting similar data acquisition/processing methods, albeit with a different focus. In this work, we aim to solve the above motivations by developing a method which will allow us to observe if an AI guitar tutor can be created.…”