mmMesh

Abstract

In this paper, we present mmMesh, the first real-time 3D human mesh estimation system using commercial portable millimeterwave devices. mmMesh is built upon a novel deep learning framework that can dynamically locate the moving subject and capture his/her body shape and pose by analyzing the 3D point cloud generated from the mmWave signals that bounce off the human body. The proposed deep learning framework addresses a series of challenges. First, it encodes a 3D human body model, which enables mmMesh to estimate complex and realistic-looking 3D human meshes from sparse point clouds. Second, it can accurately align the 3D points with their corresponding body segments despite the influence of ambient points as well as the error-prone nature and the multi-path effect of the RF signals. Third, the proposed model can infer missing body parts from the information of the previous frames. Our evaluation results on a commercial mmWave sensing testbed show that our mmMesh system can accurately localize the vertices on the human mesh with an average error of 2.47 cm. The superior experimental results demonstrate the effectiveness of our proposed human mesh construction system. CCS CONCEPTS• Human-centered computing → Ubiquitous and mobile computing; • Computer systems organization → Embedded and cyber-physical systems.

Vulnerable Road User Skeletal Pose Estimation Using mmWave Radars

“…Cao et al [20] provided a joint global-local human pose estimation network using a mmWave radar Range-Doppler Matrix (RDM) sequence, which achieves high estimation accuracy compared to using global RDM only. Xue et al [21] proposed to use the range-azimuth point cloud obtained from mmWave radar and a combination of convolutional neural network (CNN) and Long Short-Term Memory (LSTM) networks to achieve 3D human mesh model reconstruction. Zhong et al [22] provided a skeletal pose estimation method that combines point convolution to extract the local information and density from the point cloud, and they found that the accuracy of the pose estimation increased.…”

A Survey of mmWave-Based Human Sensing: Technology, Platforms and Applications

“…Additionally, it conducts the experiments under different conditions, such as poor lighting, rain, smoke, and occlusion with different materials. Similarly, mmMesh [30] chooses TI AWR1843BOOST to collect mmWave data reflected from 8 daily activities. VICON motion capture system [37] with a sampling rate of 10 fps is used to obtain high-precision dynamic pose information of the subject, which can be utilized to generate the ground truth human mesh.…”

“…Different from the existing vision, IR, and X-ray imaging systems, mmWave signals can penetrate clothes and work in low visibility conditions. Due to its higher privacy and millimeter-scale ranging resolution, mmWave-based imaging has been widely pursued for pose/posture tracking [30] [112], automatic driving [114], security concerns [65] [113] [119], etc. We summarize the related works in TABLE VIII.…”

“…Different from generating 2D silhouette images, mmMesh [30] proposes a deep learning framework to reconstruct the dynamic 3D human mesh from the mmWave signals. To alleviate the influence of ambient noise and multipath effects, mmMesh uses an attention mechanism to discriminate the point cloud and the points reflected by the subject tend to have higher qualities.…”

A Novel Spatial–Temporal Network for Gait Recognition Using Millimeter-Wave Radar Point Cloud Videos

“…Nonetheless, 4D-radar-point-cloud-based solutions still pose challenges for highperformance gait recognition. First, the limited numbers of antennas on commercial mmWave radars result in sparse radar point clouds that exhibit a lack of appearance or geometric information [19]. Second, due to the specular reflection phenomenon of mmWave signals, only parts of human body reflections propagate back to the received antennas [20,21].…”