A Wearable Device for Indoor Imminent Danger Detection and Avoidance With Region-Based Ground Segmentation

Abstract

Avoiding objects independently in indoor environments for individuals with severe visual impairment is one of the significant challenges in daily life. This paper presents a wearable application to help visually impaired people quickly build situational awareness and traverse safely. The system utilizes Red, Green, Blue, and Depth (RGB-D) camera and an Inertial Measurement Unit (IMU) to detect objects and the collision-free path in real-time. A region proposal module is presented to decide where to identify the ground from 3D point clouds. The segmented ground area can act as the traversable path, and its corresponding region in the image is removed to prevent detecting painted objects. The system can provide information about the category, distance, and direction of the detected objects by fusing the depth image and the neural network results. A 3D acoustic feedback mechanism is designed to improve the situational awareness for visually impaired people, and guild them traverse safely. The advantage of this system is that our 3D region proposal module can robustly propose the potential ground region and greatly reduce the computation cost of the ground segmentation. Besides, a typical machine-learning-based approach may miss objects because they could not be recognized, though they still may pose a danger. Another advantage of our approach is that the imminent danger detector can detect such unrecognizable objects to help users avoid a collision. Finally, experimental results demonstrate that the proposed system can be a useful indoor assistant tool to help blind individuals with collision avoidance and wayfinding.INDEX TERMS Convolutional neural network, ground segmentation, object detection, point cloud, region of interest, visual impairment, wearable assistive system.

Inertial Measurement Unit Sensors in Assistive Technologies for Visually Impaired People, a Review

“…Li et al proposed a framework to avoid objects in indoor environments [ 82 ]. This framework is composed of an RGB-D camera and IMU to detect objects and make a collision-free patch in real time.…”

“…On the other hand, in the case of [ 82 ], the initial orientation of an object was calculated by decomposing gravity from three-axis accelerations, which was represented as: …”

“…In this case, several machine learning architectures were tested, such as decision tress [ 100 ] and class labeling on computer vision [ 93 ]. Deep learning, which consists of networks that have the particularity of extracting features automatically, were applied to the rest of the articles cited in this section, of which Convolutional Neural Networks (CNN) was the most tested architecture [ 73 , 77 , 82 , 87 ]. In this review, the authors that implement inertial sensors-based IA applied the architectures to obtain the prediction of a position in 2D local coordinates (x, y) [ 99 ] and for recognition of human activities [ 67 ].…”

Smart Assistive System for Visually Impaired People Obstruction Avoidance Through Object Detection and Classification

“…Then sum of these pixels is calculated by using an integral image. Afterwards, we trained the Ada-boost classifier [12] to identify different features from the said image. This cascade classifier differentiates between face and non-face regions.…”

“…being that normal persons also ignore small things that do not intend to cause any significant harm. Therefore, the system will also recognize the scenes [2], [11], [12] for the intended user. All this information will be communicated in the form of voice to the person, in order to avoid any damage.…”