Follow the Beat? Understanding Conducting Gestures from Video

Abstract

In this paper we present a vision system that analyzes the gestures of a noted conductor conducting a real orchestra, a different approach from previous work that allowed users to conduct virtual orchestras with prerecorded scores. We use a low-resolution video sequence of a live performance of the Boston Symphony Orchestra, and we track the conductor's right hand. The tracker output is lined up with the output of an audio beat tracker run on the same sequence. The resulting analysis has numerous implications for the understanding of musical expression and gesture.

Synchronous Sympathy at the Symphony

“…Alternately, the vertical position of the conductor's arm could be obtained from video, and this data characterized over time (integrated to derive velocity and acceleration, or otherwise aggregated) to model the conductor's physical effort during the performance. A previous study made some progress toward a method to determine the vertical position of the conductor's arm by applying computer vision hand-tracking techniques to a video recording of the concert featured in this study (Nakra, Salgian, & Pfirrmann, 2009;Salgian, Pfirrmann, & Nakra, 2007). Further work is needed to extrapolate effort from the position information that was obtained.…”

Facilitating the creation of natural interactions for live audiovisual performances

“…In [15] Salgian et al presented research focusing on the analysis of music conducting gestures through the use of off-line video analysis. Camurri et al published significant research ( [2], [14]) related to real-time gesture analysis with a focus on expressive and emotional behaviour.…”