PointMapNet: Point Cloud Feature Map Network for 3D Human Action Recognition

Abstract

3D human action recognition is crucial in broad industrial application scenarios such as robotics, video surveillance, autonomous driving, or intellectual education, etc. In this paper, we present a new point cloud sequence network called PointMapNet for 3D human action recognition. In PointMapNet, two point cloud feature maps symmetrical to depth feature maps are proposed to summarize appearance and motion representations from point cloud sequences. Specifically, we first convert the point cloud frames to virtual action frames using static point cloud techniques. The virtual action frame is a 1D vector used to characterize the structural details in the point cloud frame. Then, inspired by feature map-based human action recognition on depth sequences, two point cloud feature maps are symmetrically constructed to recognize human action from the point cloud sequence, i.e., Point Cloud Appearance Map (PCAM) and Point Cloud Motion Map (PCMM). To construct PCAM, an MLP-like network architecture is designed and used to capture the spatio-temporal appearance feature of the human action in a virtual action sequence. To construct PCMM, the MLP-like network architecture is used to capture the motion feature of the human action in a virtual action difference sequence. Finally, the two point cloud feature map descriptors are concatenated and fed to a fully connected classifier for human action recognition. In order to evaluate the performance of the proposed approach, extensive experiments are conducted. The proposed method achieves impressive results on three benchmark datasets, namely NTU RGB+D 60 (89.4% cross-subject and 96.7% cross-view), UTD-MHAD (91.61%), and MSR Action3D (91.91%). The experimental results outperform existing state-of-the-art point cloud sequence classification networks, demonstrating the effectiveness of our method.

Spatio-Temporal Information Fusion and Filtration for Human Action Recognition

“…Baseline [14] 74.7% ROP [37] 86.50% DMM-HOG [10] 88.7% Actionlet ensemble [38] 88.2% HON4D [39] 88.9% DSTIP [40] 89.30% Tran et al [41] 91.9% BSC [42] 90.36% Rang-Sample Feature [43] 95.62% Kamel et al [12] 94.51% HP-DMM-CNN [12] 92.31% GMHI + GSHI + CRC [11] 94.5% Ahmad et al [44] 87.88% DDPDI [45] 96.15% Azad et al [46] 95.24% PointLSTM-late [33] 95.38% P4Transformer [34] 90.94% PointMapNet [35] 91.91% CBBMC [13] 96.3% SequentialPointNet [36] 92.6% Proposed 97.1%…”

“…Baseline [15] 66.1% 3DHOT-MBC [47] 84.4% HP-DMM-HOG [12] 73.72% DMI [18] 82.79% Yang et al [49] 88.37% DTMMN [48] 93.0% PointMapNet [35] 91.6% CBBMC [13] 94.4% Proposed 95.3%…”

“…Point cloud sequence network methods are an extension of static point cloud techniques in 3D space to 4D time-space; they directly consume spatially disordered and temporally ordered point cloud sequences to identify human actions. However, point cloud sequence network methods, like PointLSTM-late [33], P4Transformer [34], PointMapNet [35], and SequentialPointNet [36], require a large number of samples and high computational cost for model training, which hinders the application of HAR techniques in HRM scenarios.…”

“…The UTD-MHAD dataset is another widely used human action dataset. We compare the proposed method with a series of Depth-HAR methods, including Baseline [15], 3DHOT-MBC [47], HP-DMM-HOG [12], DTMMN [48], PointMapNet [35], and CBBMC [13]. The results of the comparison experiments are tabulated in Table 2.…”