Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures

Abstract

Somatosensory interaction is a kind of man-machine interfacing technique for information acquisition through human postures, which are widely used in digital art nowadays. To create opportunities for the visually impaired to enjoy digital art, two sound art systems named Dynamic Sound and Concrete Sound, which are based on somatosensory technology, were designed in this study for the visually impaired to appreciate with pleasure. The former system emphasizes resonances between humans and sound, allowing the visually impaired user to trigger different sounds by hand gestures which promote the user's physio-pleasure and ideo-pleasure. The latter system, also being controlled by hand gestures, combines sounds with three-state physical phenomena as feedbacks which are explained orally by an accompanying person to the visually impaired user, creating an inter-person communication that promotes the user's socio-pleasure. By public exhibitions, users' feedbacks were acquired via interviews, and evaluated to prove the effectiveness of the proposed systems with the following findings: 1) interactions by hand gestures offers the visually impaired with opportunities to enjoy digital art; 2) sound art provided by the systems allows the visually impaired to gain pleasure via man-machine interactions; 3) the systems innovatively integrate dynamic visual performances with auditory feedbacks in the interaction process; and 4) through the development of gesture operations, more performance techniques can be devised for sound art, allowing gesture motions to replace control interfaces in future designs.