Air-Ukulele

New Interfaces and Approaches to Machine Learning When Classifying Gestures within Music

“…Schedel and Fiebrink's [12] study on cello bow gesture recognition highlights this importance when they noted how optimisation was shown to be a factor when the performer, in the study, tried to improve model accuracy in Wekinator. Similarly, preliminary investigations by Ruan et al [25] and Dalmazzo and Ramirez [24] show that model optimisation can be beneficial when improving prediction accuracies for use in music. In terms of the impact of example length on model behaviour, the earlier noted study by Ruan et al [25] mentioned that their MPNN classification model could improve if more training data were introduced.…”

“…Tanaka et al [17] MPNN and HHMM 4 All gestures user-designed in a workshop Dalmazzo and Ramirez [24] DT and HMM 4 4 fingers (left hand, index to ring fingers) Ruan et al [25] MPNN 3 3 fingers (left hand -thumb, index, middle) Di Donato et al [26] MPNN and DTW 6 5 activation gestures and 1 modulating gesture (left arm) Erdem et al [27] RNN 6 6 gestures (exercises on guitar playing -both arms) Dalmazzo and Ramírez [28] HHMM and DT 7 Violin bowing techniques…”

“…Studies using EMG data to classify gestures within music have investigated how gestures can be used to augment instrumental practice. A study by Ruan et al [25] looked at how an 'Air-Ukulele' could be created via the use of ML to classify EMG information. The classification process used the MAV function informed by the findings made by Arief et al [29].…”

“…Scalic fingering (gesture 3): This gesture is performed by extending and isolating pressure on each individual digit of the right hand, including: thumb (i); index finger (ii); middle finger (iii); ring finger (iv); little finger (v). This gesture was created to understand and execute a nuanced detailing of EMG data, as muscle behaviour for moving individual digits is much more finely detailed than gestures activated by large and pronounced muscle groups (i.e., gesture 2) [25]. It was also chosen because literature on this topic does not use all five digits when examining such a gesture, whereas the maximum number of digits used to train a ML system is four [24,44].…”

“…The advantage of this approach is an increased mobility of piano performance; however, it also means that this approach could be further used to simulate piano performance and build novel DMIs (e.g., via interacting with 3D digital instruments). The mobile instrument approach has particular interest in the field for this reason [24,25,27]. The identification of fine EMG detailing is done, for this gesture, via post-processing and the application of a MAV function (discussed in Section 3.3.1).…”