EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution

Abstract

Purpose: Sensory-substitution devices (SSDs) provide auditory or tactile representations of visual information. These devices often generate unpleasant sensations and mostly lack color information. We present here a novel SSD aimed at addressing these issues. Methods: We developed the EyeMusic, a novel visual-to-auditory SSD for the blind, providing both shape and color information. Our design uses musical notes on a pentatonic scale generated by natural instruments to convey the visual information in a pleasant manner. A short behavioral protocol was utilized to train the blind to extract shape and color information, and test their acquired abilities. Finally, we conducted a survey and a comparison task to assess the pleasantness of the generated auditory stimuli. Results: We show that basic shape and color information can be decoded from the generated auditory stimuli. High performance levels were achieved by all participants following as little as 2-3 hours of training. Furthermore, we show that users indeed found the stimuli pleasant and potentially tolerable for prolonged use. Conclusions: The novel EyeMusic algorithm provides an intuitive and relatively pleasant way for the blind to extract shape and color information. We suggest that this might help facilitating visual rehabilitation because of the added functionality and enhanced pleasantness.

Learning New Sensorimotor Contingencies: Effects of Long-Term Use of Sensory Augmentation on the Brain and Conscious Perception

“…Already in 1969, Bach-Y-Rita and his colleagues [16] reported that the adult human brain is plastic enough for blind participants to learn how to use a tactile sensory substitution system to perceive visual input and thus recognize and localize objects in the environment. Since then, these results were supported by numerous experiments using vision-to-tactile substitution [17], or other sensory substitution devices like vision-to-auditory substitution [18,19], and vestibular-to-tactile substitution [20]. Learning how to use a sensory substitution device needs time.…”

Waist-up protection for blind individuals using the EyeCane as a primary and secondary mobility aid

“…Using the EyeCane’s distance information together with whole scene SSDs (Auvray, Hanneton, & O’Regan, 2007; Meijer, 1992), which specifically can convey visual information via audition or touch (indeed, by some definitions the EyeCane would be classified as a “minimalistic-sensory-substitution-device”) such as the EyeMusic (Abboud, Hanassy, Levy-Tzedek, Maidenbaum, & Amedi, 2014; Levy-Tzedek, Hanassy, Abboud, Maidenbaum, & Amedi, 2012), vOICe (Meijer, 1992) or BrainPort (Bach-y-Rita & W. Kercel, 2003) (see review on SSDs (Proulx et al, 2015)), will potentially enable users to integrate this information and further understand their environment. For example, both recognizing and understanding the distance to an object of interest, or understanding and recognizing obstacles as they are approached (see (Reiner, 2008) for additional potential benefits of such pairing).…”

Are Supramodality and Cross-Modal Plasticity the Yin and Yang of Brain Development? From Blindness to Rehabilitation

“…Likewise, a portion of the lateral occipital complex (LOtv) is activated in a shape recognition task using a visual-to-auditory (VASSD) or a VTSSD (Amedi et al, 2007; Ptito et al, 2012). Blind individuals can even process shape and color features by means of SSD-generated auditory stimuli (Abboud et al, 2014). Also, blind individuals recruit the visual word form area (vWFA), a specific brain region that is thought to process the visual representation of letters, when reading through a visual-to-auditory SSD (Striem-Amit et al, 2012a).…”

“…For a proper and autonomous interaction with the surrounding space, the capability to follow a specific route and avoid obstacles should also be associated with an active tracking and reaching of objects. The latter abilities have been tested in blindfolded sighted subjects while using EyeMusic (Abboud et al, 2014), a VASSD that translates the spatial location of a target into the pitch of musical notes. Levy-Tzedek et al (2012) showed that using EyeMusic, participants performed fast and accurate movements similar to those carried out with visual feedback.…”

“…More importantly, supramodal organization and crossmodal plasticity following lack of vision may both contribute to the rapid adaptation when using SSDs. On the other hand, the relationship between the proficiency in performing a specific task through the use of SSDs and the crossmodal plastic phenomena described in blind individuals is still to be fully exploited, as some authors found no behavioral differences between sighted and blind individuals (Abboud et al, 2014; Maidenbaum et al, 2014). …”