A novel tracking system for human following robots with fusion of MMW radar and monocular vision

Abstract

Purpose This paper aims to develop a robust person tracking method for human following robots. The tracking system adopts the multimodal fusion results of millimeter wave (MMW) radars and monocular cameras for perception. A prototype of human following robot is developed and evaluated by using the proposed tracking system. Design/methodology/approach Limited by angular resolution, point clouds from MMW radars are too sparse to form features for human detection. Monocular cameras can provide semantic information for objects in view, but cannot provide spatial locations. Considering the complementarity of the two sensors, a sensor fusion algorithm based on multimodal data combination is proposed to identify and localize the target person under challenging conditions. In addition, a closed-loop controller is designed for the robot to follow the target person with expected distance. Findings A series of experiments under different circumstances are carried out to validate the fusion-based tracking method. Experimental results show that the average tracking errors are around 0.1 m. It is also found that the robot can handle different situations and overcome short-term interference, continually track and follow the target person. Originality/value This paper proposed a robust tracking system with the fusion of MMW radars and cameras. Interference such as occlusion and overlapping are well handled with the help of the velocity information from the radars. Compared to other state-of-the-art plans, the sensor fusion method is cost-effective and requires no additional tags with people. Its stable performance shows good application prospects in human following robots.

Human-Following Strategy for Orchard Mobile Robot Based on the KCF-YOLO Algorithm

“…With the rapid development of computer vision technology, human-following methods based on machine vision have received widespread attention [17][18][19]. These methods involve two crucial steps: the first is target detection and tracking, and the second is human-following.…”

Four-Dimensional (4D) Millimeter Wave-Based Sensing and Its Potential Applications in Digital Construction: A Review

“…In robotics, a 4D mmw radar can be integrated as a perception sensor to enable robots to detect and track objects, relocate, and avoid obstacles in their environment [55][56][57][58], which is similar to the applications in autonomous driving. In addition, some researchers have also applied this technology to develop human-following robots [59] and estimate the velocity of robots in visually degraded environments [60].…”

A robust human target following system in corridor environment based on wall detection

“…Some scholars have used model predictive control for path tracking (Gao et al, 2022a(Gao et al, , 2023, however, they use a known trajectory. Most researchers use proportional-integraldifferential algorithm to adjust the distance and direction between robot and target (Zhu et al, 2022;Yao et al, 2021;Sarmento et al, 2022). However, due to lack of path planning module, obstacle avoidance will be difficult.…”