Wireless sensor interface and gesture-follower for music pedagogy

Abstract

We present in this paper a complete gestural interface built to support music pedagogy. The development of this prototype concerned both hardware and software components: a small wireless sensor interface including accelerometers and gyroscopes, and an analysis system enabling gesture following and recognition. A first set of experiments was conducted with teenagers in a music theory class. The preliminary results were encouraging concerning the suitability of these developments in music education. KeywordsTechnology-enhanced learning, music pedagogy, wireless interface, gesture-follower, gesture recognition INTRODUCTIONThe recent developments in the fields of movement analysis and gesture capture technology create appealing opportunities for music pedagogy. For example, traditional instruments can be augmented to provide control over digital musical processes, altering standard instrument practice and offering potentially complementary pedagogical tools. Moreover, the development of novel electronic interfaces/instruments generates even more different paradigms of music performance, giving rise to potential novel approaches in music education.In this article we present a gestural interface that was integrated in a music education context. Both hardware and software components were developed and are described here. First, we report on the design of a relatively inexpensive miniature wireless sensor system that is used with accelerometers and gyroscopes. Second, we describe a gesture analysis system programmed in the Max/MSP environment to perform gesture recognition and following. The complete prototype enables us to experiment with various pedagogical scenarios. This research is currently conducted in the framework of the European I-MAESTRO project on technology enhanced learning, focusing on music education [23].The motivation for this work is grounded in our pedagogical approach that considers physical gesture [11] as a central element for performance but also for the embodiment of music concepts and theory. Our working hypothesis is that specific gestrual interactive systems can enhance this pedagogical approach. Even if similar or complementary tools have been already proposed and carried out [7][9][10][13] [14][27], the use of digital technology and gestural interfaces in music pedagogy is at its very beginning. Any use of new technology in music education represents difficult challenges, nevertheless we believe that such an approach offers great potential. This paper is divided in three separate parts. The first two parts concern the technological developments, respectively the wireless sensor interface and the gesture follower/recognizer. In the third part, we present the pedagogical scenarios and the preliminary results we obtained after a first set of trials in music classes. WIRELESS INTERFACE AND SENSORS 2.1 RequirementsWe developed and reported previously on several wireless interfaces, that were used in applications including the augmented violin project [3] and dance performances [8]. The experienc...

KINterestTV - Towards Non–invasive Measure of User Interest While Watching TV

“…Furthermore, it comes with OpenNI software and the Primesens drivers that allow the find users in the scene and to track their virtual skeletons in 3D (see Figure 2). Among the lots of existing algorithms already used for gesture recognition (Gesture Follower [7], DTW [6], etc. ), we chose here the simplest approach: the descriptive method.…”

Continuous Realtime Gesture Following and Recognition

“…As described in [6], a HMM structure with only two possible transitions, self and next transitions can be used considering a 0 = 1/n and a 1 = 1 − 1/n, and downsampling the recorded reference by a factor n. This configuration has been used for the assessments described in section 3.4.…”

“…In this case, the learning phase is simple to operate and can be easily achieved during the normal flow of rehearsals. This approach has been iteratively designed through several specific cases [5][6][7] and implemented in a software called the Gesture Follower.…”

A Gestural Interface for Orchestral Conducting Education

“…For example, Wireless sensor interface and gesture follower (Frederic, 2007) was designed to find problems in a student's gesture compared to a teacher's gesture. Various sensors have been used in computerbased conducting systems.…”