Towards photorealistic and immersive virtual-reality environments for simulated prosthetic vision: Integrating recent breakthroughs in consumer hardware and software

Abstract

Simulated prosthetic vision (SPV) in normally sighted subjects is an established way of investigating the prospective efficacy of visual prosthesis designs in visually guided tasks such as mobility. To perform meaningful SPV mobility studies in computer-based environments, a credible representation of both the virtual scene to navigate and the experienced artificial vision has to be established. It is therefore prudent to make optimal use of existing hardware and software solutions when establishing a testing framework. The authors aimed at improving the realism and immersion of SPV by integrating state-of-the-art yet low-cost consumer technology. The feasibility of body motion tracking to control movement in photo-realistic virtual environments was evaluated in a pilot study. Five subjects were recruited and performed an obstacle avoidance and wayfinding task using either keyboard and mouse, gamepad or Kinect motion tracking. Walking speed and collisions were analyzed as basic measures for task performance. Kinect motion tracking resulted in lower performance as compared to classical input methods, yet results were more uniform across vision conditions. The chosen framework was successfully applied in a basic virtual task and is suited to realistically simulate real-world scenes under SPV in mobility research. Classical input peripherals remain a feasible and effective way of controlling the virtual movement. Motion tracking, despite its limitations and early state of implementation, is intuitive and can eliminate between-subject differences due to familiarity to established input methods.

Towards Immersive Virtual Reality Simulations of Bionic Vision

“…A major outstanding challenge is predicting what people 'see' when they use their devices. Studies of simulated prosthetic vision (SPV) often simplify phosphenes into small independent light sources [5,8,14] even though recent evidence suggests phosphenes vary drastically across subjects and electrodes [3,10]. Another challenge is addressing the narrow field of view (FOV) found in most devices (but see [11]).…”

“…This requires patients to scan the environment with head movements while trying to piece together the information [10], but many previous SPV studies are performed on computer monitors. While some studies attempt to address this [5,8,14], most fail to account for phosphene distortions. It is therefore unclear how the findings of common SPV studies would translate to real retinal prosthesis patients.…”

“…The prevailing approach to SPV is to assume that activating a grid of electrodes leads to the percept of a grid of luminous dots, the brightness of which scales linearly with stimulus amplitude [5,8,14]. By ignoring percept distortions [3,10], performance predictions of such studies can be highly misleading.…”

“…Obvious improvements could be realized by testing different electrode layouts and stimulation frequencies. Some of these factors have been modeled before in isolation, but these models often predicted higher visual acuity than what was found in clinical trials [5,6,8,14]. Therefore, using an established and psychophysically validated computational model of bionic vision may prove invaluable to generating realistic predictions of visual prosthetic performance.…”

Multimodal and alternative perception for the visually impaired: a survey

“…Table 4 lists various prosthesis devices, and simulation and processing algorithms to improve retinal prosthesis. Some recent retinal implant or related simulation algorithms are simulated prosthetic vision using photorealistic VEs (Zapf et al, 2014), peripheral visual prosthesis (Zapf et al, 2015), and bi-modal visual representation (Feng et al, 2014). Figure 10 Argus II (Ahuja and Behrend, 2013) A notable retinal prosthesis that is currently in clinical trials in the U.S. is the Argus II system ( Figure 10) by Second Sight 7 .…”