End-to-end optimization of prosthetic vision

Abstract

Neural prosthetics may provide a promising solution to restore visual perception in some forms of blindness. The restored prosthetic percept is rudimentary compared to normal vision and can be optimized with a variety of image preprocessing techniques to maximize relevant information transfer. Extracting the most useful features from a visual scene is a nontrivial task and optimal preprocessing choices strongly depend on the context. Despite rapid advancements in deep learning, research currently faces a difficult challenge in finding a general and automated preprocessing strategy that can be tailored to specific tasks or user requirements. In this paper, we present a novel deep learning approach that explicitly addresses this issue by optimizing the entire process of phosphene generation in an end-to-end fashion. The proposed model is based on a deep auto-encoder architecture and includes a highly adjustable simulation module of prosthetic vision. In computational validation experiments, we show that such an approach is able to automatically find a task-specific stimulation protocol. The results of these proof-of-principle experiments illustrate the potential of end-to-end optimization for prosthetic vision. The presented approach is highly modular and our approach could be extended to automated dynamic optimization of prosthetic vision for everyday tasks, given any specific constraints, accommodating individual requirements of the end-user.

Optimization of Neuroprosthetic Vision via End-to-end Deep Reinforcement Learning

“…4 for a comparison). Note that we do not include an explicit sparsity regularizer in our objective function as in previous work [17]. This observation suggests that optimization for a task may lead to the benefit of sparser representations in addition to providing more informative task-relevant features to the RL agent.…”

“…All convolutional layers use a kernel size of 3, with strides 1 and padding 1. We found RL performance to be more stable when using a Swish activation function after every convolution layer instead of batch normalization and leaky ReLU as in the original implementation [17]. The last convolutional layer is followed by a tanh activation, after which the outputs are scaled to the range [0, 1].…”

“…The phosphene simulator captures the perceptual effect of cortical stimulation in two steps [17]. First, the 32 × 32 stimulation pattern is upsampled to a 256 × 256 array by homogeneously inserting zeros between elements.…”

“…Such an auxiliary output, for example from a secondary depth prediction task, has been shown to improve navigation of DRL agents in complex environments [22] as well as learning saliency maps for prosthetic vision [23]. Previous work used an image reconstruction task from phosphene patterns to optimize prosthetic vision [17]. Our approach employs this image reconstruction as an auxiliary task next to an RL agent introduced here for adaptive task-dependent representation learning.…”

“…Study [15] focused on implementation of more realistic phosphene maps, while [16] employed facial landmark detection for emotion recognition. In previous work [17], an autoencoder was trained where the encoder generated stimulation patterns for simulated phosphene vision and the decoder used the phosphenes to produce a reconstruction of the sensory input. The objective function used during optimization consisted of a reconstruction loss term between original and reconstructed sensory input, and a sparsity loss term to ensure economic electrode activation.…”