Fusion of Multiple Lidars and Inertial Sensors for the Real-Time Pose Tracking of Human Motion

Abstract

Today, enhancement in sensing technology enables the use of multiple sensors to track human motion/activity precisely. Tracking human motion has various applications, such as fitness training, healthcare, rehabilitation, human-computer interaction, virtual reality, and activity recognition. Therefore, the fusion of multiple sensors creates new opportunities to develop and improve an existing system. This paper proposes a pose-tracking system by fusing multiple three-dimensional (3D) light detection and ranging (lidar) and inertial measurement unit (IMU) sensors. The initial step estimates the human skeletal parameters proportional to the target user’s height by extracting the point cloud from lidars. Next, IMUs are used to capture the orientation of each skeleton segment and estimate the respective joint positions. In the final stage, the displacement drift in the position is corrected by fusing the data from both sensors in real time. The installation setup is relatively effortless, flexible for sensor locations, and delivers results comparable to the state-of-the-art pose-tracking system. We evaluated the proposed system regarding its accuracy in the user’s height estimation, full-body joint position estimation, and reconstruction of the 3D avatar. We used a publicly available dataset for the experimental evaluation wherever possible. The results reveal that the accuracy of height and the position estimation is well within an acceptable range of ±3–5 cm. The reconstruction of the motion based on the publicly available dataset and our data is precise and realistic.

A Review: Point Cloud-Based 3D Human Joints Estimation

“…A curve skeleton expression based on the set of cross-section centroids was presented [ 54 ]. Patil et al [ 55 ] applied multiple inertial measurement unit (IMU) sensors, which were placed at the human joints to estimate the 3D position of the joints, the Lidar data compensate for displacement drift during the initial calibration of the skeleton structure. A 2.5D thinning algorithm was exerted [ 56 ], including segmentation of the occlusion region and thinning line extraction.…”

An Open-Source Platform for Human Pose Estimation and Tracking Using a Heterogeneous Multi-Sensor System

“…A unique position tracking estimation technique is achieved in the preliminary work. Since slight inclination occurs in mounting the first lidar ( in [ 31 ]) on the ceiling, the position estimation differs.…”

“…User height must be set manually for commercially available IMU sensor bodysuits for the calibration process [ 13 , 19 ], whereas the proposed approach estimates height from lidar point cloud data. The procedure for initial skeleton bone joints estimation was similar to the preliminary study approach [ 31 ], except for the height estimation.…”

“…It also shows the estimated height of a user (real height of the user is 172 cm) and skeleton construction for the data shown in Figure 3 b. Detailed skeleton parameterization and construction is given in [ 31 ].…”

“…The vector-based bone joint position and orientation estimation is given in the Algorithm 1. Figure 5 shows the vector-based pose estimation from the IMU sensors and more detail is discussed in [ 31 ]. Parallelly, the position from the lidar data was estimated by detecting the user point cloud in real-time by using a similar step as discussed above in Section 3.2 for the continuous point cloud frames.…”