VES: A Mixed-Reality System to Assist Multisensory Spatial Perception and Cognition for Blind and Visually Impaired People

Abstract

In this paper, the Virtually Enhanced Senses (VES) System is described. It is an ARCore-based, mixed-reality system meant to assist blind and visually impaired people’s navigation. VES operates in indoor and outdoor environments without any previous in-situ installation. It provides users with specific, runtime-configurable stimuli according to their pose, i.e., position and orientation, and the information of the environment recorded in a virtual replica. It implements three output data modalities: Wall-tracking assistance, acoustic compass, and a novel sensory substitution algorithm, Geometry-based Virtual Acoustic Space (GbVAS). The multimodal output of this algorithm takes advantage of natural human perception encoding of spatial data. Preliminary experiments of GbVAS have been conducted with sixteen subjects in three different scenarios, demonstrating basic orientation and mobility skills after six minutes training.

VES: A Mixed-Reality Development Platform of Navigation Systems for Blind and Visually Impaired

“…Server: A Samsung Galaxy Tab 6 tablet running a GUI interface developed with Unity game engine which renders all 3D elements of the shared virtual scenario and allows for controlling the system configuration through a GUI. Specifically, it will control the sensor input-stimuli output mapping, and the navigation environment, e.g., a previously 3D scanned room [ 16 ].…”

“…Recent mixed reality projects for BVIP resorted to walk-in-place solutions [ 11 ], some of which were based on treadmills [ 12 ], whereas others included products such as HTC Vive to capture 6-DoF movements of the user within a restricted area [ 13 , 14 ]. Conversely, Simultaneous Localization and Mapping was also used to keep track of the users’ 6-DoF head movements, emphasis placed on visual inertial odometry [ 15 , 16 ]. This solution supported outdoor/indoor scenarios without in-situ installation.…”

Inertial Measurement Unit Sensors in Assistive Technologies for Visually Impaired People, a Review

“…In addition, there are great advances in miniaturized sensors capable of providing parameters of moving objects, such as position and velocity [15][16][17]. The fusion of the advances in both sensors and artificial intelligence has led to many projects that seek to support VIP in navigation [18][19][20][21][22][23][24][25][26][27], traveling [28][29][30][31], representation of the real world [32][33][34][35], obstacle detection on wayfinding [29,[36][37][38], assistant robots [39,40], and other applications…”

“…In addition, there are great advances in miniaturized sensors capable of providing parameters of moving objects, such as position and velocity [ 15 , 16 , 17 ]. The fusion of the advances in both sensors and artificial intelligence has led to many projects that seek to support VIP in navigation [ 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ], traveling [ 28 , 29 , 30 , 31 ], representation of the real world [ 32 , 33 , 34 , 35 ], obstacle detection on wayfinding [ 29 , 36 , 37 , 38 ], assistant robots [ 39 , 40 ], and other applications for general mobility, for monitoring and improving PA, and for sports participation of the VIP [ 41 , 42 , 43 , 44 ]. The application spectrum is extensive and may include even sensor fusion for monitoring the vital signs of guide dogs in training [ 45 ].…”

Sounds That People with Visual Impairment Want to Experience

“…Sound-equipped components are also included in more advanced solutions. An example of solutions that support the functioning of people with visual impairment is a virtual sense system, which is currently under development and is designed to support the navigation of blind and visually impaired people [ 9 ]. It is intended for indoor and outdoor use and does not require prior installation in the location of intended use.…”