Auditory Coding of Visual Patterns for the Blind

Abstract

Recognition tasks of simple visual patterns have been used to assess an early visual--auditory sensory-substitution system, consisting of the coupling of a rough model of the human retina with an inverse model of the cochlea, by means of a pixel-frequency relationship. The potential advantage of the device, compared with previous ones, is to give the blind the ability to both localise and recognise visual patterns. Four evaluation sessions assessed the performance of twenty-four blindfolded sighted subjects using the device. Subjects had to recognise twenty-five visual patterns, one at a time, using a head-mounted small camera and interpreting the corresponding sounds given by the device. Half the subjects were trained by means of a correction feedback procedure during ten one-hour training sessions embedded in between the evaluation sessions. Results revealed extremely successful training effects. Performance of trained subjects significantly increased with practice compared with the untrained control group. The improvement was also observed for new patterns, demonstrating a learning-process generalisation. The negative correlation observed between scores and processing time showed that the subjects' response accuracy was related to their speed. In conclusion, simple pattern recognition is possible with a fairly natural vision-to-audition coding scheme, given the possibility for the subjects to have sensory--motor interactions while using the device.

Perception With Compensatory Devices: From Sensory Substitution to Sensorimotor Extension

“…Furthermore, in order to enhance the similarity with the human visual system, the receptor field of the PSVA has a higher resolution in the center of the picture. Studies that have been conducted with auditory devices have shown the possibility of object localization (Auvray et al, 2007a;Renier et al, 2005a) and form recognition (Arno et al, 1999;Arno et al, 2001 with the PSVA; Cronly- Dillon et al, 1999Dillon et al, , 2000Pollok, Schnitzler, Mierdorf, Stoerig, & Schnitzler, 2005 with the Voice). Interestingly, recent studies have also demonstrated the possibility of recreating visual illusions with the PSVA (Renier et al, 2005b).…”

Vertical-horizontal illusion present for sighted but not early blind humans using auditory substitution of vision

“…A quarter of the stimuli were the same as those used in the VHI experiment. The evaluation procedure was identical to the one used by Arno et al (1999). After each stimulus exploration, subjects re-created their observations with a set of metal bars.…”

“…Before performing the experiments, sighted and early blind subjects were trained to use the PSVA in a patternrecognition task. This training phase (adapted from Arno, Capelle, Wanet-Defalque, Catalan-Ahumada, & Veraart, 1999) consisted of five 1-h sessions preceded and followed by an evaluation session. The sighted subjects worked blindfolded.…”

“…A score ranging from 0 to 100 was then assigned, based on how well the re-creation matched the stimulus pattern. After the training sessions, geometrical figures of equivalent complexity to the VHI figures (Arno et al, 1999) were tested. Early blind subjects obtained a mean score of 80.34% correct responses (SD 14.88) and blindfolded sighted subjects obtained a mean score of 77.21% (SD 13.72) [F(1,28) 1, p .5714].…”

The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms

“…The PVSA (Prosthesis Substituting Vision for Audition) uses pitch to encode position in both the horizontal and vertical axes. The PVSA also implements a bias inspired by the foveal region of the human eye, which dedicates more "space" to pixels in the centre of the field of view (Arno, Capelle, Wanet-Defalque, Catalan-Ahumada, & Veraart, 1999). An alternative approach is to sonify only those pixels that represent edges rather than surfaces.…”