Interactive tangible user interface for music learning

Abstract

Abstract² This paper introduced muSurface, an interactive surface with ability to interact with tangible musical symbols. Children can explore musical symbols by placing the symbols on the virtual musical staffs displayed on the playing surface, and then the system plays the resulting melody. The system senses musical symbols on the surface using the rear diffused illumination technique, and displays the processed output back via a projector and speakers. The main underlying techniques are image classification based on K-nearest neighbor classifier using 9 features derived from musical symbols. The distance transform also used to find locations of musical notes in order to determine the pitches of the notes. Index Terms--augmented reality; human computer interaction; machine vision; musical instrument digital interfaces; object recognitionI.

Solfeggio learning and the influence of a mobile application based on visual, auditory and tactile modalities

“…In his work, users could play music by selecting cards on a table using a head‐mounted display. While AR was used in music applications, like in Brown () or Waranusast, Bang‐ngoen, and Thipakorn (), it was not used to recognise or play a song according to music notation. In mySolfeggio, the user can point their video camera at the music notation and use the AR tool (Vuforia, ) in order to recognise images of the music notation, which are built into the mobile application in the form of AR‐markers.…”

Explaining multi-threaded task scheduling using tangible user interfaces in higher educational contexts

“…By allowing children to touch and manipulate tangible musical notes, the work in [29] reported significant progress through the "boring" phases of learning music. Moving tokens were used in [30] to familiarise students with astrological notions of moon phases and the motion of planets.…”

Interactive Rhythm Learning System by Combining Tablet Computers and Robots

“…To improve children's learning motivation, interactive systems that involve the music instruments with electronic techniques are developed for learning music in recent years [3][4][5][6][7][8][9][10][11][12][13][14]. Bae et al developed "Drum On" to reduce boredom and other limitations [3].…”

“…This application helps children learn music through interesting and engaging games. Waranusast et al developed "muSurface", an interactive surface with ability to interact with tangible musical symbols [11]. Users place tangible music symbols on the virtual musical staffs displayed on the playing surface, and muSurface plays the resulting melody according to the symbols.…”