Deep Learning-Based Violin Bowing Action Recognition

Abstract

We propose a violin bowing action recognition system that can accurately recognize distinct bowing actions in classical violin performance. This system can recognize bowing actions by analyzing signals from a depth camera and from inertial sensors that are worn by a violinist. The contribution of this study is threefold: (1) a dataset comprising violin bowing actions was constructed from data captured by a depth camera and multiple inertial sensors; (2) data augmentation was achieved for depth-frame data through rotation in three-dimensional world coordinates and for inertial sensing data through yaw, pitch, and roll angle transformations; and, (3) bowing action classifiers were trained using different modalities, to compensate for the strengths and weaknesses of each modality, based on deep learning methods with a decision-level fusion process. In experiments, large external motions and subtle local motions produced from violin bow manipulations were both accurately recognized by the proposed system (average accuracy > 80%).

A Motion Capture Data-Driven Automatic Labanotation Generation Model Using the Convolutional Neural Network Algorithm

“…As can be seen from Figure 11, the bone features designed in this study have achieved the best recognition results, reaching about 94% recognition accuracy, which is about 2% higher than the features extracted from literature [28], which fully proves the effectiveness of the bone node vector features proposed in this study.…”

“…Displacement, velocity, and acceleration can reflect the posture information of human body, and angle can reflect the rotation information of human body. e included angle of knee joint is taken as an example (the included angle of elbow joint is the same), as shown in Figure 3: If the leftUpLeg-node world coordinate system is (x 0 , y 0 , z 0 ), leftLowLeg-node world coordinate system is (x 1 , y 1 , z 1 ), and leftFoot-node world coordinate system is (x 2 , y 2 , z 2 ), the side length of the triangle formed by these three nodes of the human leg and the included angle of the knee joint can be calculated as follows: [22], target detection [23], behavior recognition [24], and natural language processing and other fields [25][26][27]and has made extraordinary achievements [28].…”

“…e recognition accuracy of human upper-limb movements is first compared to a feature of normalized human orientation extracted from literature [28] using the same spatial segmentation method, as shown in Figure 11. Fea-ture1 denotes the original action data, Feature2 the features added with normalization of human body orientation, Method in reference [19] Method in reference [20] Method in reference [21] CNN method…”

Ensemble of Multimodal Deep Learning Models for Violin Bowing Techniques Classification

“…Dalmazzo and Ramirez [11] presented bowing technique classification by applying Hierarchical Markov Model (HHMM) to data from inertial sensors and audio recordings acquired from a single violinist playing a simple G-Major scale and the accuracy obtained with motion only, audio only and audio + motion features are 93.2%, 39.01% and 94.61%, respectively. Sun et al [12] presented the use of Deep learning models for classifying violin bowing techniques by analyzing the signals from inertial sensors and depth camera and were able to get an average accuracy greater than 80%. Dalmazzo and Ramirez [13] presented Deep Learning techniques for classifying violin bowing techniques such as detaché , legato, martelé , collé , staccato, ricochet, tré molo and col legno and were able to get an accuracy of 97.15%, 98.55%, 99.23% with CNN, 3D-MultiHeaded CNN, CNN LSTM models respectively.…”

“…The previous studies have presented methods that incorporate either motion data [11][12][13][14] or audio data [10] and dataset considered was from a simple playing of violin such as scale. To the best of our knowledge, there is no work available where both motion and audio features are considered for classifying bowing categories.…”