Deep Learning-Based Object Detection, Localisation and Tracking for Smart Wheelchair Healthcare Mobility

Abstract

This paper deals with the development of an Advanced Driver Assistance System (ADAS) for a smart electric wheelchair in order to improve the autonomy of disabled people. Our use case, built from a formal clinical study, is based on the detection, depth estimation, localization and tracking of objects in wheelchair’s indoor environment, namely: door and door handles. The aim of this work is to provide a perception layer to the wheelchair, enabling this way the detection of these keypoints in its immediate surrounding, and constructing of a short lifespan semantic map. Firstly, we present an adaptation of the YOLOv3 object detection algorithm to our use case. Then, we present our depth estimation approach using an Intel RealSense camera. Finally, as a third and last step of our approach, we present our 3D object tracking approach based on the SORT algorithm. In order to validate all the developments, we have carried out different experiments in a controlled indoor environment. Detection, distance estimation and object tracking are experimented using our own dataset, which includes doors and door handles.

Deep learning model for doors detection: A contribution for context-awareness recognition of patients with Parkinson’s disease

“…Door detection approaches have been extensively developed for indoor robot navigation with social, assistance or domestic applications (Banerjee et al, 2015;Borgsen et al, 2014;Chen et al, 2014;Dai et al, 2013;Derry & Argall, 2013;Fernández-Caramés et al, 2014;He & Zhu, 2017;Kakillioglu et al, 2016;Lecrosnier et al, 2021;Llopart et al, 2017;Othman & Rad, 2020;Quintana et al, 2018;Ramoa et al, 2020;Sekkal et al, 2013;Shalaby et al, 2014;Spournias et al, 2020;Tian et al, 2013;Yuan et al, 2016). Robotic wheelchairs, humanoids, or systems for aid persons with visual impairments were other target fields of research in door detection algorithms (Derry & Argall, 2013;He & Zhu, 2017;Lecrosnier et al, 2021;Llopart et al, 2017;Othman & Rad, 2020;Ramoa et al, 2020;Shalaby et al, 2014;Tian et al, 2013). Table 1 summarizes the state-of-the-art survey in computer vision-based systems for door detection.…”

“…This problem was solved by using the power of transfer learning with a pre-trained model using the TF Object Detection API (Adrian Rosebrock, 2017). Doors' detection using DL has already been addressed for indoor robot navigation with social, assistance or domestic applications, as more detailed in section 2 (Banerjee, Long, Du, Polido, Feng, Atkeson, Gennert, & Padir, 2015;Borgsen, Schöpfer, Ziegler, & Wachsmuth, 2014;Chen, Qu, Zhou, Weng, Wang, & Fu, 2014;Dai et al, 2013;Derry & Argall, 2013;Fernández-Caramés, Moreno, Curto, Rodríguez-Aragón, & Serrano, 2014;He & Zhu, 2017;Kakillioglu, Ozcan, & Velipasalar, 2016;Lecrosnier et al, 2021;Llopart, Ravn, & Andersen, 2017;Othman & Rad, 2020;Quintana, Prieto, Adán, & Bosché, 2018;Ramoa, Alexandre, & Mogo, 2020;Sekkal, Pasteau, Babel, Brun, & Leplumey, 2013;Shalaby, Salem, Khamis, & Melgani, 2014;Spournias, Antonopoulos, Keramidas, Voros, & Stojanovic, 2020;Tian, Yang, Yi, & Arditi, 2013;Yuan, Hashim, Zaki, & Huddin, 2016). However, these studies did not address PD problem, a contribution introduced by this work.…”

Smart Wheelchairs: A Review on Control Methods

“…YOLO (You Only Look Once) is another pre-trained computer vision model that relatively popular due to its simplicity and processing speed. An Advanced Driver Assistance System (ADAS) [10] for a smart wheelchair is presented based on object detection, depth estimation, localization and tracking in indoor environment where it used YOLOv3 algorithm used for object detection and SORT algorithm used for 3D object tracking. Another research uses another variation of YOLO for bus detection and recognize open and closed bus door for automated boarding [11].…”

A Real-Time Distance Prediction via Deep Learning and Microsoft Kinect

“…On the other hand, the 3D detection techniques launch a third dimension that discloses more detailed information of an object's size and location [3]. The useful depth information in 3D helps to analyse the real environment effectively [4,5]. For an instance, RGBD (Red, Green, Blue and Depth) camera can capture the surroundings perfectly as compared to a normal camera because it consists of a depth camera which able to extract the depth images.…”