Deep reinforcement learning for task-based feature learning in prosthetic vision

White, Jack; Kameneva, Tatiana; McCarthy, Chris

doi:10.1109/embc.2019.8856541

Cited by 11 publications

(8 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Future work could extend the current approach with other or more complex tasks. For instance, with reinforcement learning strategies (see White et al., 2019 ), the model could be extended to perform tasks that more closely related to the everyday actions that need to be performed by the end-user, such as object manipulation ( Levine, Finn, Darrell, & Abbeel, 2015 ) or object avoidance ( LeCun, Muller, Ben, Cosatto, & Flepp, 2005 ).…”

Section: Discussionmentioning

confidence: 99%

“…Second, as a result of practical, medical, or biophysical limitations of the neural interface, one might want to tailor the stimulation parameters to additional constraints. Recent work on task-based feature learning for prosthetic vision suggests that deep learning models can be used to overcome such issues ( White, Kameneva, & McCarthy, 2019 ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

End-to-end optimization of prosthetic vision

et al. 2022

View full text Add to dashboard Cite

Neural prosthetics may provide a promising solution to restore visual perception in some forms of blindness. The restored prosthetic percept is rudimentary compared to normal vision and can be optimized with a variety of image preprocessing techniques to maximize relevant information transfer. Extracting the most useful features from a visual scene is a nontrivial task and optimal preprocessing choices strongly depend on the context. Despite rapid advancements in deep learning, research currently faces a difficult challenge in finding a general and automated preprocessing strategy that can be tailored to specific tasks or user requirements. In this paper, we present a novel deep learning approach that explicitly addresses this issue by optimizing the entire process of phosphene generation in an end-to-end fashion. The proposed model is based on a deep auto-encoder architecture and includes a highly adjustable simulation module of prosthetic vision. In computational validation experiments, we show that such an approach is able to automatically find a task-specific stimulation protocol. The results of these proof-of-principle experiments illustrate the potential of end-to-end optimization for prosthetic vision. The presented approach is highly modular and our approach could be extended to automated dynamic optimization of prosthetic vision for everyday tasks, given any specific constraints, accommodating individual requirements of the end-user.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

End-to-end optimization of prosthetic vision

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Some researchers also built 3D virtual scenes for obstacle avoidance experiments. 56 Tables 1 and 2 summarize more details of the above studies in terms of algorithms and simulation experiments.…”

Section: Methodsmentioning

confidence: 99%

“…Moreover, the sparse computation of unary potentials improved the speed of semantic labeling to ensure that it could run on wearable auxiliary devices in real time. White et al 56 focused on basic orientation and mobility and adopted the deep reinforcement learning (DRL) algorithm to learn visual features by completing visual tasks. To solve the DRL navigation problem in a 3D environment, a new model of learning visual features through task-based simulation was proposed by using an improved version of Asynchronous Advantage Act-Critic (A3C) algorithm, 57 and the learned features were directly converted into real RGB-D images.…”

Section: Methodsmentioning

confidence: 99%

The application of computer vision to visual prosthesis

et al. 2021

View full text Add to dashboard Cite

A visual prosthesis is an auxiliary device for patients with blinding diseases that cannot be treated with conventional surgery or drugs. It converts captured images into corresponding electrical stimulation patterns, according to which phosphenes are generated through the action of internal electrodes on the visual pathway to form visual perception. However, due to some restrictions such as the few implantable electrodes that the biological tissue can accommodate, the induced perception is far from ideal. Therefore, an important issue in visual prosthesis research is how to detect and present useful information in low-resolution prosthetic vision to improve the visual function of the wearer. In recent years, with the development and broad application of computer vision methods, researchers have investigated the possibility of their utilization in visual prostheses by simulating prosthetic visual percepts.Through the optimization of visual perception by image processing, the efficiency of visual prosthesis devices can be further improved to better meet the needs of prosthesis wearers. In this article, recent works on prosthetic vision centering on implementing computer vision methods are reviewed. Differences, strengths, and weaknesses of the mentioned methods are discussed. The development directions of optimizing prosthetic vision and improving methods of visual perception are analyzed.

show abstract

“…One way to address this issue is to include an auxiliary output unrelated to the main task [21]. Such an auxiliary output, for example from a secondary depth prediction task, has been shown to improve navigation of DRL agents in complex environments [22] as well as learning saliency maps for prosthetic vision [23]. Previous work used an image reconstruction task from phosphene patterns to optimize prosthetic vision [17].…”

Section: Related Workmentioning

confidence: 99%

Optimization of Neuroprosthetic Vision via End-to-end Deep Reinforcement Learning

Küçükoğlu

Rueckauer

Ahmad

et al. 2022

Preprint

View full text Add to dashboard Cite

Visual neuroprostheses are a promising approach to restore basic sight in visually impaired people. A major challenge is to condense the sensory information contained in a complex environment into meaningful stimulation patterns at low spatial and temporal resolution. Previous approaches considered task-agnostic feature extractors such as edge detectors or semantic segmentation, which are likely suboptimal for specific tasks in complex dynamic environments. As an alternative approach, we propose to optimize stimulation patterns by end-to-end training of a feature extractor using deep reinforcement learning agents in virtual environments. We present a task-oriented evaluation framework to compare different stimulus generation mechanisms, such as static edge-based and adaptive end-to-end approaches like the one introduced here. Our experiments in Atari games show that stimulation patterns obtained via task-dependent end-to-end optimized reinforcement learning result in equivalent or improved performance compared to fixed feature extractors on high difficulty levels. These findings signify the relevance of adaptive reinforcement learning for neuroprosthetic vision in complex environments.

show abstract

Deep reinforcement learning for task-based feature learning in prosthetic vision

Cited by 11 publications

References 6 publications

End-to-end optimization of prosthetic vision

End-to-end optimization of prosthetic vision

The application of computer vision to visual prosthesis

Optimization of Neuroprosthetic Vision via End-to-end Deep Reinforcement Learning

Contact Info

Product

Resources

About