New Interfaces and Approaches to Machine Learning When Classifying Gestures within Music

Abstract

Interactive music uses wearable sensors (i.e., gestural interfaces—GIs) and biometric datasets to reinvent traditional human–computer interaction and enhance music composition. In recent years, machine learning (ML) has been important for the artform. This is because ML helps process complex biometric datasets from GIs when predicting musical actions (termed performance gestures). ML allows musicians to create novel interactions with digital media. Wekinator is a popular ML software amongst artists, allowing users to train models through demonstration. It is built on the Waikato Environment for Knowledge Analysis (WEKA) framework, which is used to build supervised predictive models. Previous research has used biometric data from GIs to train specific ML models. However, previous research does not inform optimum ML model choice, within music, or compare model performance. Wekinator offers several ML models. Thus, we used Wekinator and the Myo armband GI and study three performance gestures for piano practice to solve this problem. Using these, we trained all models in Wekinator and investigated their accuracy, how gesture representation affects model accuracy and if optimisation can arise. Results show that neural networks are the strongest continuous classifiers, mapping behaviour differs amongst continuous models, optimisation can occur and gesture representation disparately affects model mapping behaviour; impacting music practice.

Investigation on the Sampling Frequency and Channel Number for Force Myography Based Hand Gesture Recognition

“…The study focuses on static actions, so only the signal during the maintain period is segmented for hand motion classification. The main reason is that the signal during start-up period and recovery period has large variation, which would decrease the overall classification accuracy [ 33 ]. Besides, the correct outputs during static period can quickly overwhelm incorrect ones during these transient stages, because there are very limited samples during transient stages.…”

Classifying Biometric Data for Musical Interaction Within Virtual Reality

“…Studies comparing ML models for gesture recognition in music, using biometric datasets, with Wekinator are sparse [23]. A study [9] compared two ML models when classifying four violin performance gestures (five including one nonmusical gesture), using a Decision Tree (DT) J48 (via Wekinator) and a Hidden Markov Model (HMM), plus a Myo GI (using IMU and EMG data) to build an 'air violin'.…”

“…Data parameters are then post-processed by scaling and mapping the data for DSP and ML. Data is scaled by applying an absolute value function to each EMG parameter (reducing its polarity) and then scaling it between 0-100, as described in another study [23]. However, this work builds on this method by using the minima and maxima recorded EMG inputs by the Myo user (first author) when performing our studied gestures (see Sect.…”

“…Performance gestures can be represented both dynamically and statically when using ML in music practice [23], which is problematic. This is because the gestural signal onset can be either omitted or included in the ML model training process and affect musical intention.…”

“…This is because the gestural signal onset can be either omitted or included in the ML model training process and affect musical intention. Previous research has looked at the efficacy of ML models in Wekinator on two musical gestures used in piano practice [22] and has extended this investigation with further gestures, including model optimisation, plus the effect of including gestural signal onset on model accuracy/efficacy [23]. Our study is informed by this prior research.…”