Assisted Piano Pedagogy through 3D Visualization of Piano Playing

Mora, Javier; Lee, Won-Sook; Comeau, Gilles; Shirmohammadi, Shervin; Saddik, A. El

doi:10.1109/have.2006.283791

Cited by 14 publications

(3 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Notably, most systems still largely focus on the score and the associated errors, with feedback related to muscle activation (Montes et al, 1993) or posture (Mora et al, 2006) being more rare. The overall focus on correctness of notes rather than overall musicianship and expression has been criticized by Xiao and Ishii (2016).…”

Section: Achieving Movement Fluencymentioning

confidence: 99%

Characterizing Movement Fluency in Musical Performance: Toward a Generic Measure for Technology Enhanced Learning

et al. 2019

View full text Add to dashboard Cite

Virtuosity in music performance is often associated with fast, precise, and efficient sound-producing movements. The generation of such highly skilled movements involves complex joint and muscle control by the central nervous system, and depends on the ability to anticipate, segment, and coarticulate motor elements, all within the biomechanical constraints of the human body. When successful, such motor skill should lead to what we characterize as fluency in musical performance. Detecting typical features of fluency could be very useful for technology-enhanced learning systems, assisting and supporting students during their individual practice sessions by giving feedback and helping them to adopt sustainable movement patterns. In this study, we propose to assess fluency in musical performance as the ability to smoothly and efficiently coordinate while accurately performing slow, transitionary, and rapid movements. To this end, the movements of three cello players and three drummers at different levels of skill were recorded with an optical motion capture system, while a wireless electromyography (EMG) system recorded the corresponding muscle activity from relevant landmarks. We analyzed the kinematic and coarticulation characteristics of these recordings separately and then propose a combined model of fluency in musical performance predicting music sophistication. Results suggest that expert performers' movements are characterized by consistently smooth strokes and scaling of muscle phasic coactivation. The explored model of fluency as a function of movement smoothness and coarticulation patterns was shown to be limited by the sample size, but it serves as a proof of concept. Results from this study show the potential of a technology-enhanced objective measure of fluency in musical performance, which could lead to improved practices for aspiring musicians, instructors, and researchers.

show abstract

Section: Achieving Movement Fluencymentioning

confidence: 99%

Characterizing Movement Fluency in Musical Performance: Toward a Generic Measure for Technology Enhanced Learning

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Rendering of 3D shapes has many applications. For example, this can be used for designing therapeutic interfaces [6], interactive pedagogy [5,12], shape learning by kindergarten students [4], learning art and music [32], virtual and augmented reality applications [17] etc. Users can perform rendering and transformation of 3D shapes by means of interactive participation through natural gestures.…”

Section: Related Workmentioning

confidence: 99%

Visual rendering of shapes on 2D display devices guided by hand gestures

2019

View full text Add to dashboard Cite

Designing of touchless user interface is gaining popularity in various contexts. Using such interfaces, users can interact with electronic devices even when the hands are dirty or nonconductive. Also, user with partial physical disability can interact with electronic devices using such systems. Research in this direction has got major boost because of the emergence of low-cost sensors such as Leap Motion, Kinect or RealSense devices. In this paper, we propose a Leap Motion controller-based methodology to facilitate rendering of 2D and 3D shapes on display devices. The proposed method tracks finger movements while users perform natural gestures within the field of view of the sensor. In the next phase, trajectories are analyzed to extract extended Npen++ features in 3D. These features represent finger movements during the gestures and they are fed to unidirectional left-to-right Hidden Markov Model (HMM) for training. A one-to-one mapping between gestures and shapes is proposed. Finally, shapes corresponding to these gestures are rendered over the display using MuPad interface. We have created a dataset of 5400 samples recorded by 10 volunteers. Our dataset contains 18 geometric and 18 non-geometric shapes such as "circle", "rectangle", "flower", "cone", "sphere" etc. The proposed methodology achieves an accuracy of 92.87% when evaluated using 5-fold cross validation method. Our experiments revel that the extended 3D features perform better than existing 3D features in the context of shape representation and classification. The method can be used for developing useful HCI applications for smart display devices.

show abstract

“…3D motion capture has been used in [10] to assist in piano pedagogy by visualising the posture of a professional performer and comparing it to that of the student. It has been featured in a double bass instruction DVD [11] to help illustrate elements of bowing technique.…”

Section: Introductionmentioning

confidence: 99%