Classification of the Sidewalk Condition Using Self-Supervised Transfer Learning for Wheelchair Safety Driving

Abstract

The demand for wheelchairs has increased recently as the population of the elderly and patients with disorders increases. However, society still pays less attention to infrastructure that can threaten the wheelchair user, such as sidewalks with cracks/potholes. Although various studies have been proposed to recognize such challenges, they mainly depend on RGB images or IMU sensors, which are sensitive to outdoor conditions such as low illumination, bad weather, and unavoidable vibrations, resulting in unsatisfactory and unstable performance. In this paper, we introduce a novel system based on various convolutional neural networks (CNNs) to automatically classify the condition of sidewalks using images captured with depth and infrared modalities. Moreover, we compare the performance of training CNNs from scratch and the transfer learning approach, where the weights learned from the natural image domain (e.g., ImageNet) are fine-tuned to the depth and infrared image domain. In particular, we propose applying the ResNet-152 model pre-trained with self-supervised learning during transfer learning to leverage better image representations. Performance evaluation on the classification of the sidewalk condition was conducted with 100% and 10% of training data. The experimental results validate the effectiveness and feasibility of the proposed approach and bring future research directions.

Deep-Learning-Based Approach for Automated Detection of Irregular Walking Surfaces for Walkability Assessment with Wearable Sensor

“…Numerous advanced deep-learning-based approaches have been developed for the automated evaluation of roadways or sidewalks utilizing infrastructure-based data such as images, videos or GIS technologies [26][27][28][29][30][31][32]. In one study [33], acceleration data from smartphones mounted on vehicles were used to train deep learning models for road surface monitoring and pothole detection.…”

Design of an Assisted Driving System for Obstacle Avoidance Based on Reinforcement Learning Applied to Electrified Wheelchairs

“…Convolutional neural networks (CNNs) have been employed in various attempts, leveraging their effectiveness in image processing. Specifically, CNNs excel in visual obstacle detection, yielding successful outcomes in this context [78][79][80][81][82][83][84]. When integrated into a larger software ecosystem, their outputs can serve as inputs to traditional motion planners, enabling the computation of a path that avoids the detected obstacles.…”

A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications

“…One of the solutions to address the lack of training data is employing the pre-trained models of ImageNet for the target task. For some applications, this type of TL from ImageNet has significantly improved the results compared with training from scratch [257,258]. However, for some other applications such as medical imaging applications, this type of TL from ImageNet does not help to address the issue of lack of training data.…”