Air violin: a machine learning approach to fingering gesture recognition

Dalmazzo, David; Ramírez, Rafael

doi:10.1145/3139513.3139526

Cited by 21 publications

(27 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…When compared with everyday actions, violin bowing is characterized by subtle motions. Dalmazzo and Ramirez [ 10 ] formulated a method for recognizing such subtle motions; in this method, electromyography (EMG) signals from the violinist’s forearm are analyzed for finger gesture recognition. Another study classified bowing gestures by applying machine learning [ 11 ] to data from inertial sensors and audio recordings.…”

Section: Related Workmentioning

confidence: 99%

“…Another study classified bowing gestures by applying machine learning [ 11 ] to data from inertial sensors and audio recordings. The shortcoming of the first study lies in the unreliability of EMG; EMG signals [ 10 ] vary depending on which part of the forearm the sensor is located. The shortcoming of the second study lies in its small dataset: the data were gathered from one performer who repeatedly executed only a limited variety of bowing motions [ 11 ].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Deep Learning-Based Violin Bowing Action Recognition

Sun

Liu

Chang

2020

Sensors

View full text Add to dashboard Cite

We propose a violin bowing action recognition system that can accurately recognize distinct bowing actions in classical violin performance. This system can recognize bowing actions by analyzing signals from a depth camera and from inertial sensors that are worn by a violinist. The contribution of this study is threefold: (1) a dataset comprising violin bowing actions was constructed from data captured by a depth camera and multiple inertial sensors; (2) data augmentation was achieved for depth-frame data through rotation in three-dimensional world coordinates and for inertial sensing data through yaw, pitch, and roll angle transformations; and, (3) bowing action classifiers were trained using different modalities, to compensate for the strengths and weaknesses of each modality, based on deep learning methods with a decision-level fusion process. In experiments, large external motions and subtle local motions produced from violin bow manipulations were both accurately recognized by the proposed system (average accuracy > 80%).

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Deep Learning-Based Violin Bowing Action Recognition

Sun

Liu

Chang

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…These systems perform audio analysis and give feedback on whether the learner is producing the right note, tempo, phrasing and similar. Other systems extend this by providing feedback on the movements that produce the notes, such as fingering in violin playing (Dalmazzo and Ramirez, 2017) or the hand used in drumming (Kanke et al, 2017). In addition to the score based exercises, some systems employ haptic actuators to indicate which limbs or fingers the student should move (Holland et al, 2010; Lee and Choi, 2014), even providing “passive learning” of movement sequences (Huang et al, 2008).…”

Section: Achieving Movement Fluencymentioning

confidence: 99%

Characterizing Movement Fluency in Musical Performance: Toward a Generic Measure for Technology Enhanced Learning

et al. 2019

View full text Add to dashboard Cite

Virtuosity in music performance is often associated with fast, precise, and efficient sound-producing movements. The generation of such highly skilled movements involves complex joint and muscle control by the central nervous system, and depends on the ability to anticipate, segment, and coarticulate motor elements, all within the biomechanical constraints of the human body. When successful, such motor skill should lead to what we characterize as fluency in musical performance. Detecting typical features of fluency could be very useful for technology-enhanced learning systems, assisting and supporting students during their individual practice sessions by giving feedback and helping them to adopt sustainable movement patterns. In this study, we propose to assess fluency in musical performance as the ability to smoothly and efficiently coordinate while accurately performing slow, transitionary, and rapid movements. To this end, the movements of three cello players and three drummers at different levels of skill were recorded with an optical motion capture system, while a wireless electromyography (EMG) system recorded the corresponding muscle activity from relevant landmarks. We analyzed the kinematic and coarticulation characteristics of these recordings separately and then propose a combined model of fluency in musical performance predicting music sophistication. Results suggest that expert performers' movements are characterized by consistently smooth strokes and scaling of muscle phasic coactivation. The explored model of fluency as a function of movement smoothness and coarticulation patterns was shown to be limited by the sample size, but it serves as a proof of concept. Results from this study show the potential of a technology-enhanced objective measure of fluency in musical performance, which could lead to improved practices for aspiring musicians, instructors, and researchers.

show abstract

“…Par défaut, 5 gestes sont reconnus, mais par l'intermédiaire de son SDK, de nouveaux gestes peuvent être appris. Cet appareil est utilisé dans plusieurs travaux relativement récents comme [10,11] pour la performance musicale (violonistes), pour la classification du mouvement des doigts par [32], pour l'analyse de la navigation par geste de la main par [24], pour la réalité virtuelle par [22], pour la création d'un mapping interactif entre les gestes et des sons musicaux [34] etc. Le Myo™ offre plusieurs avantages par rapport aux autres appareils : un coût abordable (environ 250 €), une configuration simple, et la possibilité d'être caché ce qui est très intéressant en terme de performances artistiques [35].…”

Section: Reconnaissance Des Gestes Des Mainsunclassified