Deep Learning Reader for Visually Impaired

Abstract

Recent advances in machine and deep learning algorithms and enhanced computational capabilities have revolutionized healthcare and medicine. Nowadays, research on assistive technology has benefited from such advances in creating visual substitution for visual impairment. Several obstacles exist for people with visual impairment in reading printed text which is normally substituted with a pattern-based display known as Braille. Over the past decade, more wearable and embedded assistive devices and solutions were created for people with visual impairment to facilitate the reading of texts. However, assistive tools for comprehending the embedded meaning in images or objects are still limited. In this paper, we present a Deep Learning approach for people with visual impairment that addresses the aforementioned issue with a voice-based form to represent and illustrate images embedded in printed texts. The proposed system is divided into three phases: collecting input images, extracting features for training the deep learning model, and evaluating performance. The proposed approach leverages deep learning algorithms; namely, Convolutional Neural Network (CNN), Long Short Term Memory (LSTM), for extracting salient features, captioning images, and converting written text to speech. The Convolution Neural Network (CNN) is implemented for detecting features from the printed image and its associated caption. The Long Short-Term Memory (LSTM) network is used as a captioning tool to describe the detected text from images. The identified captions and detected text is converted into voice message to the user via Text-To-Speech API. The proposed CNN-LSTM model is investigated using various network architectures, namely, GoogleNet, AlexNet, ResNet, SqueezeNet, and VGG16. The empirical results conclude that the CNN-LSTM based training model with ResNet architecture achieved the highest prediction accuracy of an image caption of 83%.

Alpha-Numeric Character Recognition for Blinds Using Deep-Learning Techniques

“…LITERATURE SURVEY A remedy for the issue for visually impaired to help with their difficulties in reading texts in non-ideal conditions that may include changing lighting conditions, the different font styles used in the document and the direction of the characters. The proposed system [1], developed using the MATLAB platform, serves as an affordable substitute, making it accessible to a broader user base. Its core functionalities rely on two key components: Optical Character Recognition (OCR) and Text-to-Speech technology.…”

A Novel Algorithm of Distance Calculation Based-on Grid-Edge-Depth-Map and Gyroscope for Visually-Impaired

“…The GED map size of the block is 25 pixels [10]. The depth map was built based on a disparity map by calculating the camera baseline and focal length [13,18,19]. Then it was divided by x for every pixel of the disparity map.…”

Image Captioning for the Visually Impaired and Blind: A Recipe for Low-Resource Languages

“…For instance, verbal description of a scene acquired in real time can help visually impaired and blind people to determine their location, navigate their surroundings, access visual information, feel safer, and increase their situational awareness [11], [12]. Indeed, there are some recent works looking at the use of image captioning to help the visually impaired and blind [13], [14]. These studies focus on the English language, because benchmark image captioning datasets with human-generated captions (e.g., Microsoft COCO Captions dataset [15]) are generally created in this high-resource language.…”