Immersive Virtual Reality Simulations of Bionic Vision

Kasowski, Justin; Beyeler, Michael

doi:10.1145/3519391.3522752

Cited by 10 publications

(5 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interestingly, both encoders do slightly worse with very small radial current spread (ρ), suggesting that some amount of phosphene overlap is needed to build shapes. This does align with some previous behavioural results [25], and may have implications for the design of future visual prostheses such as [35], which attempt to minimize current spread altogether.…”

Section: Discussionsupporting

confidence: 85%

A Hybrid Neural Autoencoder for Sensory Neuroprostheses and Its Applications in Bionic Vision

Granley¹,

Relic²,

Beyeler³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Sensory neuroprostheses are emerging as a promising technology to restore lost sensory function or augment human capacities. However, sensations elicited by current devices often appear artificial and distorted. Although current models can often predict the neural or perceptual response to an electrical stimulus, an optimal stimulation strategy solves the inverse problem: what is the required stimulus to produce a desired response? Here we frame this as an end-to-end optimization problem, where a deep neural network encoder is trained to invert a known, fixed forward model that approximates the underlying biological system. As a proof of concept, we demonstrate the effectiveness of our hybrid neural autoencoder (HNA) on the use case of visual neuroprostheses. We found that HNA is able to produce high-fidelity stimuli from the MNIST and COCO datasets that outperform conventional encoding strategies and surrogate techniques across all tested conditions. Overall this is an important step towards the long-standing challenge of restoring high-quality vision to people living with incurable blindness and may prove a promising solution for a variety of neuroprosthetic technologies.Preprint. Under review.

show abstract

Section: Discussionsupporting

confidence: 85%

A Hybrid Neural Autoencoder for Sensory Neuroprostheses and Its Applications in Bionic Vision

Granley¹,

Relic²,

Beyeler³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The implementation from [38] includes some of these models, but it only works by offline rendering a single image or video, taking a long time to process. Additionally, this implementation is adapted for a VR simulator of prosthetic vision in [39]. However, it only introduces the spatial model without temporal dynamics, and the image processing is mostly reduced to edge detection.…”

Section: Related Workmentioning

confidence: 99%

RASPV: A Robotics Framework for Augmented Simulated Prosthetic Vision

Perez-Yus,

Santos-Villafranca,

Tomas-Barba

et al. 2024

IEEE Access

View full text Add to dashboard Cite

One of the main challenges of visual prostheses is to augment the perceived information to improve the experience of its wearers. Given the limited access to implanted patients, in order to facilitate the experimentation of new techniques, this is often evaluated via Simulated Prosthetic Vision (SPV) with sighted people. In this work, we introduce a novel SPV framework and implementation that presents major advantages with respect to previous approaches. First, it is integrated into a robotics framework, which allows us to benefit from a wide range of methods and algorithms from the field (e.g. object recognition, SLAM, obstacle avoidance, autonomous navigation, deep learning). Second, we go beyond traditional image processing with 3D point clouds processing using an RGB-D camera, allowing us to robustly detect the floor, obstacles and the structure of the scene. Third, it works either with a real camera or in a virtual environment, which gives us endless possibilities for immersive experimentation through a head-mounted display. Fourth, we incorporate a validated temporal phosphene model that replicates time effects into the generation of visual stimuli. Finally, we have proposed, developed and tested several applications within this framework, such as avoiding moving obstacles, providing a general understanding of the scene, staircase detection, helping the subject to navigate an unfamiliar space and reach a destination, and object and person detection. We provide experimental results in real and virtual environments. The code will be publicly available at www.github.com/aperezyus/RASPV

show abstract

“…Lastly, a number of studies focused on spatial cognition tasks, such as obstacle avoidance ( Zapf, Boon, Lovell, & Suaning, 2015 , 2016 ; Endo et al., 2019 ) and wayfinding ( Vergnieux, Macé, & Jouffrais, 2014 ). By design, these tasks require a more immersive setup that allows for the incorporation of head and eye movements as well as locomotion ( Kasowski & Beyeler, 2022 ). Most of these studies incorporated a fully immersive design for their task, although a few used VR headsets but required their subjects to sit/stand in place and instead use a keyboard/controller to move.…”

Section: Research Areasmentioning

confidence: 99%

A systematic review of extended reality (XR) for understanding and augmenting vision loss

et al. 2023

Self Cite

View full text Add to dashboard Cite

Over the past decade, extended reality (XR) has emerged as an assistive technology not only to augment residual vision of people losing their sight but also to study the rudimentary vision restored to blind people by a visual neuroprosthesis. A defining quality of these XR technologies is their ability to update the stimulus based on the user’s eye, head, or body movements. To make the best use of these emerging technologies, it is valuable and timely to understand the state of this research and identify any shortcomings that are present. Here we present a systematic literature review of 227 publications from 106 different venues assessing the potential of XR technology to further visual accessibility. In contrast to other reviews, we sample studies from multiple scientific disciplines, focus on technology that augments a person’s residual vision, and require studies to feature a quantitative evaluation with appropriate end users. We summarize prominent findings from different XR research areas, show how the landscape has changed over the past decade, and identify scientific gaps in the literature. Specifically, we highlight the need for real-world validation, the broadening of end-user participation, and a more nuanced understanding of the usability of different XR-based accessibility aids.

show abstract

Immersive Virtual Reality Simulations of Bionic Vision

Cited by 10 publications

References 50 publications

A Hybrid Neural Autoencoder for Sensory Neuroprostheses and Its Applications in Bionic Vision

A Hybrid Neural Autoencoder for Sensory Neuroprostheses and Its Applications in Bionic Vision

RASPV: A Robotics Framework for Augmented Simulated Prosthetic Vision

A systematic review of extended reality (XR) for understanding and augmenting vision loss

Contact Info

Product

Resources

About