Seeing with Sound? Exploring Different Characteristics of a Visual-to-Auditory Sensory Substitution Device

Abstract

Sensory substitution devices convert live visual images into auditory signals, for example with a web camera (to record the images), a computer (to perform the conversion) and headphones (to listen to the sounds). In a series of three experiments, the performance of one such device (`The vOICe') was assessed under various conditions on blindfolded sighted participants. The main task that we used involved identifying and locating objects placed on a table by holding a webcam (like a flashlight) or wearing it on the head (like a miner's light). Identifying objects on a table was easier with a hand-held device, but locating the objects was easier with a headmounted device. Brightness converted into loudness was less effective than the reverse contrast (dark being loud), suggesting that performance under these conditions (natural indoor lighting, novice users) is related more to the properties of the auditory signal (ie the amount of noise in it) than the cross-modal association between loudness and brightness. Individual differences in musical memory (detecting pitch changes in two sequences of notes) was related to the time taken to identify or recognise objects, but individual differences in self-reported vividness of visual imagery did not reliably predict performance across the experiments. In general, the results suggest that the auditory characteristics of the device may be more important for initial learning than visual associations.

Improving training for sensory augmentation using the science of expertise

“…Although a few basic training tasks might be sufficient to use the BrainPort device (Wood et al, 2009), other groups trained participants with the device for three hours or more (Polat & Uneri, 2010;Uneri & Polat, 2009;Barros et al, 2010). Likewise, while a few hours training or more (Arno et al, 1999;Auvray et al, 2007) might be considered necessary for even basic navigational tasks with more complex devices, some studies provided participants with little or no familiarisation before testing (Brown et al, 2011).…”

The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms

“…In order for the participant to link an auditory component (e.g. a high pitch sound in the second time point) to an external location/object that can be acted upon they must also "embody" the device itself (O'Regan, 1992;Brown et al, 2011). For instance if a camera is worn on the head, then the position in space that the sound denotes is determined by the current orientation of the head in addition to the properties of the sound itself.…”

“…headphones, vibrotactile array). Within a few hours of training novice participants have some ability to localise and recognise objects (Auvray, Hanneton, & O'Regan, 2007;Brown, Macpherson, & Ward, 2011) and generalise to new objects (Kim & Zatorre, 2008). Expert blind users recruit 'visual' cortices to process the substituted sense (Amedi et al, 2007;Merabet et al, 2009;Poirier, De Volder, Tranduy, & Scheiber, 2007; cf.…”

“…More recently, Brown et al (2011) explored different settings within the "vOICe" device; for instance, comparing contrast settings (bright=loud v. bright=quiet) and the length of the soundscape (1s v. 2s) in a 2x2 design. This is one of the first attempts to determine the optimal parameters for perceiving sonified images in a sensory substitution device, but the number of parameters that can be varied in a given experiment are very low.…”

"I Always Wanted to See the Night Sky"

“…Only aesthetic ratings of two devices have been done previously by BVIPs [3]. Here The vOICe has examining obje nformational c brain [4,5,7,8 vOICe has bee on android de reported to be B widely adopted availability sin adopted it for l visual rehabilit appear to pers research has fo ittle attention h…”