A Google Glass Based Real-Time Scene Analysis for the Visually Impaired

Abstract

Blind and Visually Impaired People (BVIP) are likely to experience difficulties with tasks that involve scene recognition. Wearable technology has played a significant role in researching and evaluating systems developed for and with the BVIP community. This paper presents a system based on Google Glass designed to assist BVIP with scene recognition tasks, thereby using it as a visual assistant. The camera embedded in the smart glasses is used to capture the image of the surroundings, which is analyzed using the Custom Vision Application Programming Interface (Vision API) from Azure Cognitive Services by Microsoft. The output of the Vision API is converted to speech, which is heard by the BVIP user wearing the Google Glass. A dataset of 5000 newly annotated images is created to improve the performance of the scene description task in Indian scenarios. The Vision API is trained and tested on this dataset, increasing the mean Average Precision (mAP) score from 63% to 84%, with an IoU > 0.5. The overall response time of the proposed application was measured to be less than 1 second, thereby providing accurate results in realtime. A Likert scale analysis was performed with the help of the BVIP teachers and students at the "Roman Catherine Lobo School for the Visually Impaired" at Mangalore, Karnataka, India. From their response, it can be concluded that the application helps the BVIP better recognize their surrounding environment in real-time, proving the device effective as a potential assistant for the BVIP.

Intelligent Environments and Assistive Technologies For Assisting Visually Impairment People: A Systematic Literature Review

“…Frequency Reference T1 11 articles [46], [106], [51], [97], [109], [27], [54], [110], [82], [55], [114] T2 25 articles [32], [29], [62], [24], [34], [73], [48], [50], [39], [94], [40], [95], [42], [78], [67], [31], [101], [37], [103], [85], [38], [111], [112], [113], [115] T3 18 articles [86], [87], [88], [90], [71], [64], [93], [35], [72], [74], [36], [96], [98], [69], …”

“…Rao et al [86] outlines a real-time scene analysis system based on Google Glass to assist visually impaired people with scene recognition tasks. This system uses Microsoft's Vision API from Azure Cognitive Services to process images and converts the API output into speech, which the user hears.…”

“…Additionally, the lack of necessary skills, poor public sector access, and prohibitive costs further hinder the adoption of assistive technologies. Rao et al [86] highlighted that achieving realtime performance with a smart-glass is challenging, as the complexity of the algorithm directly impacts the device's response time, unless the algorithm runs on a powerful machine.…”

IBGS: A Wearable Smart System to Assist Visually Challenged

“…The smart crutches proposed by A.S. Romadhon et al use an ultrasonic sensor and a water sensor, and additionally add a heart sensor to detect the user's health status [12]. Smartphones have very powerful features and excellent operating systems, and many researches are devoted to using smartphones to obtain better functions [13]- [19]. S.…”

“…This work contains a dataset of 5000 images, which enables excellent scene recognition. It should not be overlooked that the average price of Google Glass is $1500 [19]. There are also many studies trying to reduce costs as much as possible while providing outdoor walking assistance for the blind [20]- [22].…”