Smartphone Drum: Gesture-based Digital Musical Instruments Application for Deaf and Hard of Hearing People

Abstract

Smartphone applications that allow users to enjoy playing musical instruments have emerged, opening up numerous related opportunities. However, it is difficult for deaf and hard of hearing (DHH) people to use these apps because of limited access to auditory information. When using real instruments, DHH people typically feel the music from the vibrations transmitted by the instruments or the movements of the body, which is not possible when playing with these apps. We introduce "smartphone drum, " a smartphone application that presents a drum-like vibrotactile sensation when the user makes a drumming motion in the air with their smartphone like a drumstick. We implemented an early prototype and received feedback from six DHH participants. We discuss the technical implementation and the future of new instruments of vibration.

Analysis of Accessible Digital Musical Instruments through the lens of disability models: a case study with instruments targeting d/Deaf people

“…Smartphone Drum (Iijima et al, 2021) is a smartphone app that presents a vibrotactile sensation similar to that of a drum when the musician makes a drumming motion in the air with their smartphone as a drumstick. d/Deaf people are not included in the design process, but they are included in the evaluation process.…”

The Role of Haptics in Training and Games for Hearing-Impaired Individuals: A Systematic Review

“…Despite the interest from industry and academia in video games equipped with vibrotactile feedback [70], the literature reporting their application to enhance the experience for the hearing-impaired population appears to be sparse. In our research, only three articles directly addressed gamification aspects in their design processes [46,47,62]. Cano et al [46] focused on a table game for children aged 7 to 11 with hearing impairment.…”

“…However, the latter is somewhat limited, offering a buzz-like sensation only when a child's answer is incorrect, given its secondary role since the smartphone screen simultaneously displays a corresponding sad face emoticon. The second paper [47] introduces a mobile app that offers vibrotactile feedback in response to a detected drumming gesture by the smartphone. The interaction and feedback are described clearly, but the study lacks emphasis on the gaming aspect, even if the keyword game is included.…”

“…Moving to the device choice, we can observe that almost every article adopts a unique approach. The most common device is the smartphone [38,44,47,54,58], given its nearubiquity; with most people owning one or at least being familiar with it, smartphones serve as convenient and portable tools for training and enhancing experiences. However, the compact size of smartphones comes with some drawbacks, particularly concerning vibrotactile performance.…”

SHITARA: Sending Haptic Induced Touchable Alarm by Ring-shaped Air vortex

“…Finally, other approaches outside of the DHH's daily life can also be found about haptic presentation for the DHH. In the musical scene, "Smartphone Drum [22]" is a smartphone application that plays with haptic, presenting a drum-like vibrotactile sensation. In addition, some studies [23,24] have replaced the start signal in athletics with a haptic presentation in the sports scene.…”