Stereosonic vision: Exploring visual-to-auditory sensory substitution mappings in an immersive virtual reality navigation paradigm

Abstract

Sighted people predominantly use vision to navigate spaces, and sight loss has negative consequences for independent navigation and mobility. The recent proliferation of devices that can extract 3D spatial information from visual scenes opens up the possibility of using such mobility-relevant information to assist blind and visually impaired people by presenting this information through modalities other than vision. In this work, we present two new methods for encoding visual scenes using spatial audio: simulated echolocation and distance-dependent hum volume modulation. We implemented both methods in a virtual reality (VR) environment and tested them using a 3D motion-tracking device. This allowed participants to physically walk through virtual mobility scenarios, generating data on real locomotion behaviour. Blindfolded sighted participants completed two tasks: maze navigation and obstacle avoidance. Results were measured against a visual baseline in which participants performed the same two tasks without blindfolds. Task completion time, speed and number of collisions were used as indicators of successful navigation, with additional metrics exploring detailed dynamics of performance. In both tasks, participants were able to navigate using only audio information after minimal instruction. While participants were 65% slower using audio compared to the visual baseline, they reduced their audio navigation time by an average 21% over just 6 trials. Hum volume modulation proved over 20% faster than simulated echolocation in both mobility scenarios, and participants also showed the greatest improvement with this sonification method. Nevertheless, we do speculate that simulated echolocation remains worth exploring as it provides more spatial detail and could therefore be more useful in more complex environments. The fact that participants were intuitively able to successfully navigate space with two new visual-to-audio mappings for conveying spatial information motivates the further exploration of these and other mappings with the goal of assisting blind and visually impaired individuals with independent mobility.

VES: A Mixed-Reality System to Assist Multisensory Spatial Perception and Cognition for Blind and Visually Impaired People

“…Several projects in this line of work implement sensory substitution interfaces using periodic measurements of the user's position and body movement [24]. Simultaneous Locating and Mapping technology (SLAM) from Google Tango eased the development of interfaces for BVI people's navigation, which required no additional infrastructure installation [25,26]. This can be also applied to systems such as Google ARCore [27], which capture the motion of compatible smartphones through Concurrent Odometry and Mapping technology (COM).…”

Audio Guide for Visually Impaired People Based on Combination of Stereo Vision and Musical Tones

“…Massiceti and her coauthors developed an inland navigation model based on the relationship between visual perceptions and a sound relative to the position indicating obstacles or a target [16]. The authors extracted 3D visual spatial information to enable the mobility of blind and visually impaired people.…”

“…HEYA et al, 2018 [10] SLAM KNN Sound KITT et al, 2010 [11] Proximity Method Kalman filter Visual XUE et al, 2016 [12] Proximity Method RANSAC Visual PRESTI et al, 2019 [15] Proximity Method Weighted average Polytonic MASSICETI et al, 2018 [16] Proximity Method KNN humming sound BUJACZ et al, 2016 [17] Proximity Method Particle filter humming sound ALCANTARILLA et al, 2012 [14] SLAM Weighted average Visual CHEN et al, 2014 [6] PDR Kalman filter Visual…”

Research on Optimization Method of VR Task Scenario Resources Driven by User Cognitive Needs

“…In the virtual reality (VR) system task scenario, the mapping relationship analysis between the visual expression of multi-channel information resources and the user's hidden cognitive needs is an important part of studying the user's cognitive load and user experience. At the same time, it is very important for designers to predict a user's cognitive load and satisfaction during the system construction process [1][2][3]. In the field of human-computer interaction, in order to accurately grasp the level of user experience perception, many scholars have provided a valuable research basis for the analysis of user cognitive behavior and satisfaction.…”

“…The two elements are of equal importance compared to each other. 3 Compared with the two elements, the former is slightly more important than the latter. 5…”