Semantic and structural image segmentation for prosthetic vision

Abstract

Prosthetic vision is being applied to partially recover the retinal stimulation of visually impaired people. However, the phosphenic images produced by the implants have very limited information bandwidth due to the poor resolution and lack of color or contrast. The ability of object recognition and scene understanding in real environments is severely restricted for prosthetic users. Computer vision can play a key role to overcome the limitations and to optimize the visual information in the prosthetic vision, improving the amount of information that is presented. We present a new approach to build a schematic representation of indoor environments for simulated phosphene images. The proposed method combines a variety of convolutional neural networks for extracting and conveying relevant information about the scene such as structural informative edges of the environment and silhouettes of segmented objects. Experiments were conducted with normal sighted subjects with a Simulated Prosthetic Vision system. The results show good accuracy for object recognition and room identification tasks for indoor scenes using the proposed approach, compared to other image processing methods.

Adaptive kernel selection network with attention constraint for surgical instrument classification

“…Moreover, Zhao et al [47] propose a visual tracking approach using the CNN with a spatial transformer network and a spatiotemporal context learning algorithm for the process of tool tracking frame by frame, which is devoted to enhancing the context-awareness of surgeons in the operating room. Sanchez-Garcia et al [35] present a new CNN-based fusion approach to build a schematic representation of indoor environments for simulated phosgene images, which aims to train and partially recover the retinal stimulation of visually impaired people in rehabilitation training.…”

Toward Long-Term Communication With the Brain in the Blind by Intracortical Stimulation: Challenges and Future Prospects

“…Despite these encouraging results, most visual prosthesis devices only try to emulate the phototransducer aspects of the retina and do not consider the complex processes that are found in the mammalian visual system. Some researchers have proposed that performance could be increased significantly by incorporating neural code (Nirenberg and Pandarinath, 2012), whereas others promote the use of computer vision algorithms and techniques of artificial intelligence (Sanchez-Garcia et al, 2020). Although more studies are still needed, we expect that bio-inspired visual encoders based on intelligent signal and image-processing strategies, together with new cutting-edge artificial intelligence algorithms running neuromorphic hardware, could have a significant impact in the future to facilitate the interpretation of the processed signals (Fernandez, 2018).…”

End-to-end optimization of prosthetic vision

“…The phosphene encodings that were found by the model in this condition show similarities to those in ref. [37], who demonstrated that pre-processing with semantic segmentation may successfully improve object recognition performance in simulated prosthetic vision (compared to pre-processing with conventional edge detection techniques). Note, that in the proposed end-to-end architecture, supervised training was merely applied to the output reconstructions and that the labels do not directly control the phosphene representations themselves.…”