Real-Time Musical Conducting Gesture Recognition Based on a Dynamic Time Warping Classifier Using a Single-Depth Camera

Abstract

Gesture recognition is a human−computer interaction method, which is widely used for educational, medical, and entertainment purposes. Humans also use gestures to communicate with each other, and musical conducting uses gestures in this way. In musical conducting, conductors wave their hands to control the speed and strength of the music played. However, beginners may have a limited comprehension of the gestures and might not be able to properly follow the ensembles. Therefore, this paper proposes a real-time musical conducting gesture recognition system to help music players improve their performance. We used a single-depth camera to capture image inputs and establish a real-time dynamic gesture recognition system. The Kinect software development kit created a skeleton model by capturing the palm position. Different palm gestures were collected to develop training templates for musical conducting. The dynamic time warping algorithm was applied to recognize the different conducting gestures at various conducting speeds, thereby achieving real-time dynamic musical conducting gesture recognition. In the experiment, we used 5600 examples of three basic types of musical conducting gestures, including seven capturing angles and five performing speeds for evaluation. The experimental result showed that the average accuracy was 89.17% in 30 frames per second.

Enhancing spatial perception through sound: mapping human movements into MIDI

“…Their target is to extend gestures input to a 3D visual representation of the control space in which one can efficiently apply multiple parameter variation techniques by carrying out movements. In [8], the authors illustrate a real-time musical conducting gesture recognition system that supports music players in enhancing their performance. They used a single-depth camera to capture image inputs and to establish a real-time gesture recognition system.…”

Perceiving space through sound: mapping human movements into MIDI

“…The MATRIX interface manipulates the parameters of a synthesis engine or effect algorithm in real-time, in response to the performer's expressive gestures. In [10], authors illustrate a real-time musical conducting gesture recognition system that supports music players in enhancing their performance. They used a single-depth camera to capture image inputs and to establish a real-time gesture recognition system.…”

Translation Resilient Opportunistic WiFi Sensing

“…Using WiFi signal for human sensing has gained much attention over the past decade. Compared to other technologies such as cameras [3], [4] and wearable sensors [5], [6], WiFi based sensing technology is considered as an ideal solution for future smart homes and healthcare residences since it is not intrusive, provides a wider coverage, is not sensitive to lighting conditions and the users do not need to wear any uncomfortable devices, making this technology completely passive.…”