LidSonic for Visually Impaired: Green Machine Learning-Based Assistive Smart Glasses with Smart App and Arduino

Abstract

Smart wearable technologies such as fitness trackers are creating many new opportunities to improve the quality of life for everyone. It is usually impossible for visually impaired people to orientate themselves in large spaces and navigate an unfamiliar area without external assistance. The design space for assistive technologies for the visually impaired is complex, involving many design parameters including reliability, transparent object detection, handsfree operations, high-speed real-time operations, low battery usage, low computation and memory requirements, ensuring that it is lightweight, and price affordability. State-of-the-art visually impaired devices lack maturity, and they do not fully meet user satisfaction, thus more effort is required to bring innovation to this field. In this work, we develop a pair of smart glasses called LidSonic that uses machine learning, LiDAR, and ultrasonic sensors to identify obstacles. The LidSonic system comprises an Arduino Uno device located in the smart glasses and a smartphone app that communicates data using Bluetooth. Arduino collects data, manages the sensors on smart glasses, detects objects using simple data processing, and provides buzzer warnings to visually impaired users. The smartphone app receives data from Arduino, detects and identifies objects in the spatial environment, and provides verbal feedback about the object to the user. Compared to image processing-based glasses, LidSonic requires much less processing time and energy to classify objects using simple LiDAR data containing 45-integer readings. We provide a detailed description of the system hardware and software design, and its evaluation using nine machine learning algorithms. The data for the training and validation of machine learning models are collected from real spatial environments. We developed the complete LidSonic system using off-the-shelf inexpensive sensors and a microcontroller board costing less than USD 80. The intention is to provide a design of an inexpensive, miniature, green device that can be built into, or mounted on, any pair of glasses or even a wheelchair to help the visually impaired. This work is expected to open new directions for smart glasses design using open software tools and off-the-shelf hardware.

Requirements, Challenges, and Use of Digital Devices and Apps for Blind and Visually Impaired

“…Among the most important topics that blind people use applications for are currency recognition, color recognition, object recognition, navigation applications, and virtual assistance. However, image processing methods' computational cost and power consumption present performance concern with portable or wearable low-power devices [12].…”

“…In contrast, wearables and white canes are used in commercial solutions. Solutions in application development are either dependent on humans, in which the visually impaired communicate with someone who can help them recognize and navigate their environment, or they are standalone, non-external support software with solutions that do not require external human assistance [12]. Assistive technologies have emerged to help people with poor vision or vision impairment preserve their independence and safety.…”

“…Assistive Technology for visually impaired users is projected to expand quickly and have a significant influence on the lives of people and the elderly in ways that have never been feasible previously. We conducted an extensive review (see [12]) on the assistive technologies for visually impaired and found an array of solutions for visually-impaired and blind people continually emerging in the recent times that provide different functionalities and capabilities aimed at a wide spectrum of visual impairments.…”

“…The information from this survey along with a detailed literature review of academic and commercial technologies for the visually impaired was used to establish the research gap, design requirements, and a comprehensive understanding of the relevant landscape. This information in turn was used to develop LidSonic V1.0, a pair of smart glasses called LidSonic that uses machine learning, LiDAR, and ultrasonic sensors to identify obstacles; see [12].…”

LidSonic V2.0: LiDAR and Deep Learning-based Green Assistive Edge Device to Enhance Mobility for the Visually Impaired

“…The work presented in this paper is a substantial extension of our earlier system Lid-Sonic (V1.0) [39]. LidSonic V2.0, the new version of the system, uses both machine learning and deep learning methods for classification, as opposed to V1.0 which uses machine learning alone.…”