Deep Learning Based Wearable Assistive System for Visually Impaired People

Abstract

In this paper, we propose a deep learning based assistive system to improve the environment perception experience of visually impaired (VI). The system is composed of a wearable terminal equipped with an RGBD camera and an earphone, a powerful processor mainly for deep learning inferences and a smart phone for touch-based interaction. A data-driven learning approach is proposed to predict safe and reliable walkable instructions using RGBD data and the established semantic map. This map is also used to help VI understand their 3D surrounding objects and layout through well-designed touchscreen interactions. The quantitative and qualitative experimental results show that our learning based obstacle avoidance approach achieves excellent results in both indoor and outdoor datasets with lowlying obstacles. Meanwhile, user studies have also been carried out in various scenarios and showed the improvement of VI's environment perception experience with our system.

A Systematic Review of Urban Navigation Systems for Visually Impaired People

“…DeepLabV3 is a semantic segmentation used to define 15 obstacles, such as a sidewalk, pole, building [ 60 ]. FuseNet generated semantic images to use with RGB and RGB-D images to provide walkable instructions for the user [ 44 ]. Duh et al [ 39 ] and Yang et al [ 47 ] used semantic segmentation to recognize obstacles.…”

“…The number of obstacles defines the number of covered objects in each dataset whether they were applied for a BVIP use case or not. As shown in the table, there is no dataset that defines all needed obstacles from BVIP’s perspectives [ 44 , 60 , 68 ]. Although Lin et al [ 44 ] built a dataset with 6000 obstacles for BVIP’s usage, this dataset contains only low-lying obstacles.…”

“…As shown in the table, there is no dataset that defines all needed obstacles from BVIP’s perspectives [ 44 , 60 , 68 ]. Although Lin et al [ 44 ] built a dataset with 6000 obstacles for BVIP’s usage, this dataset contains only low-lying obstacles. This table underlines the need to build a new dataset from BVIP perspective that cover objects on different levels.…”

“…For example, if there is a tree branch alongside the sidewalk, what action should the BVIP take? We suggest using a method to evaluate the situation (level of hazard) [ 45 ], then generating a compatible instruction [ 44 ]. Additionally, we suggest an obstacle avoidance system that depends on sensors to continuously detect obstacles and to use a camera from time to time where a scene description is needed, so as reduce consuming power.…”

Helping the Blind to Get through COVID-19: Social Distancing Assistant Using Real-Time Semantic Segmentation on RGB-D Video

“…Lin et al [ 37 ] developed a wearable assistive system by generating collision-free instructions with touchscreen interactions to fully make use of semantic segmentation maps. In [ 38 , 39 ], instance-specific semantic segmentation was leveraged to help blind people to recognize objects in their surroundings by using state-of-the-art instance segmentation models such as Mask R-CNN [ 40 ].…”