Efficiency of Sensory Substitution Devices Alone and in Combination With Self-Motion for Spatial Navigation in Sighted and Visually Impaired

Abstract

Human adults can optimally combine vision with self-motion to facilitate navigation. In the absence of visual input (e.g., dark environments and visual impairments), sensory substitution devices (SSDs), such as The vOICe or BrainPort, which translate visual information into auditory or tactile information, could be used to increase navigation precision when integrated together or with self-motion. In Experiment 1, we compared and assessed together The vOICe and BrainPort in aerial maps task performed by a group of sighted participants. In Experiment 2, we examined whether sighted individuals and a group of visually impaired (VI) individuals could benefit from using The vOICe, with and without self-motion, to accurately navigate a three-dimensional (3D) environment. In both studies, 3D motion tracking data were used to determine the level of precision with which participants performed two different tasks (an egocentric and an allocentric task) and three different conditions (two unisensory conditions and one multisensory condition). In Experiment 1, we found no benefit of using the devices together. In Experiment 2, the sighted performance during The vOICe was almost as good as that for self-motion despite a short training period, although we found no benefit (reduction in variability) of using The vOICe and self-motion in combination compared to the two in isolation. In contrast, the group of VI participants did benefit from combining The vOICe and self-motion despite the low number of trials. Finally, while both groups became more accurate in their use of The vOICe with increased trials, only the VI group showed an increased level of accuracy in the combined condition. Our findings highlight how exploiting non-visual multisensory integration to develop new assistive technologies could be key to help blind and VI persons, especially due to their difficulty in attaining allocentric information.

A self-training program for sensory substitution devices

“…Although use of tactile cues (e.g. [ 26 ]) alongside auditory cues, thus creating an audio-tactile multisensory experience would be optimal as they can be used also by blind individuals (see for instance the work of Jicol and colleagues showing advantages towards the combined use of auditory and tactile cues[ 27 ]), the use of such settings have their drawbacks. First, the use of tactile information has a lower resolution than auditory and visual cues (tactile bandwidth 100 bits per second [ 28 ], audition bandwidth 104 bits per second [ 29 ], visual bandwidth of 4.3*106 bits per second [ 30 ]).…”

Brain-Machine Interfaces to Assist the Blind

“… Striem-Amit et al (2012b) evaluated the audio-visual acuity of the vOICe users after receiving 73 h of training with the device, more than half of whom had attained a visual acuity of 20/320 which outclasses the threshold of blindness (20/400) defined by the World Health organization. Moreover, several studies with the vOICe demonstrated that blind individuals can learn to identify geometric forms and shapes ( Amedi et al, 2007 ), read ( Striem-Amit et al, 2012a ), locate objects in space ( Auvray et al, 2007 ) and even learn virtual maps ( Jicol et al, 2020 ). Since then, a new version of the vOICe has been developed to add color information to the mixture of the visual information given by the device.…”

Multisensory inclusive design with sensory substitution

“…The live camera feed captures an aerial map of multiple targets and delivers this information to the users in tactile, sonification or tactile-sonification forms. Here, the user can acquire spatial information necessary for navigation via electro-tactile stimulation on her tongue and also via sonifications which are delivered by bone conduction headphones (Jicol et al, 2020 ). The cross-modal display prototype here applies the principles of spatial and temporal coincidence by aligning the sensory information available to the camera.…”