Graph and Temporal Convolutional Networks for 3D Multi-person Pose Estimation in Monocular Videos

Abstract

Despite the recent progress, 3D multi-person pose estimation from monocular videos is still challenging due to the commonly encountered problem of missing information caused by occlusion, partially out-of-frame target persons, and inaccurate person detection. To tackle this problem, we propose a novel framework integrating graph convolutional networks (GCNs) and temporal convolutional networks (TCNs) to robustly estimate camera-centric multi-person 3D poses that does not require camera parameters. In particular, we introduce a human-joint GCN, which unlike the existing GCN, is based on a directed graph that employs the 2D pose estimator's confidence scores to improve the pose estimation results. We also introduce a human-bone GCN, which models the bone connections and provides more information beyond human joints. The two GCNs work together to estimate the spatial frame-wise 3D poses and can make use of both visible joint and bone information in the target frame to estimate the occluded or missing human-part information. To further refine the 3D pose estimation, we use our temporal convolutional networks (TCNs) to enforce the temporal and human-dynamics constraints. We use a joint-TCN to estimate person-centric 3D poses across frames, and propose a velocity-TCN to estimate the speed of 3D joints to ensure the consistency of the 3D pose estimation in consecutive frames. Finally, to estimate the 3D human poses for multiple persons, we propose a root-TCN that estimates camera-centric 3D poses without requiring camera parameters. Quantitative and qualitative evaluations demonstrate the effectiveness of the proposed method.

Deep Neural Networks in Video Human Action Recognition: A review

“…As per our verification, most of the vision data could be converted into the format of embedding and compared through contrastive learning. Meanwhile, the research work [156]- [159] discussed how deep learning works well in skeleton-based behavior recognition through the common frameworks with GCN and CNN. The double stream of combined score results has been verified its advantages in multiple behavior recognition tasks with double joints stream in research [159], and the combination of multiple models in deep feature extractions [156].…”

“…Meanwhile, the research work [156]- [159] discussed how deep learning works well in skeleton-based behavior recognition through the common frameworks with GCN and CNN. The double stream of combined score results has been verified its advantages in multiple behavior recognition tasks with double joints stream in research [159], and the combination of multiple models in deep feature extractions [156]. The multi-modality human behavior recognition has also proven its availability in the increased identification accuracy.…”

Deep Neural Networks in Video Human Action Recognition: A review

“…As our verification, most of vision data could be converted into the format of embedding and compared through contrastive learning. Meanwhile, the research work [122]- [125] discussed how the deep learning works well in the skeleton-based behavior recognition through the common frameworks with GCN and CNN. The double stream of combined score results have been verified its advantages in multiple behavior recognition tasks with double joints stream in research [125], and multiple models of deep feature extractions [122].…”

“…Meanwhile, the research work [122]- [125] discussed how the deep learning works well in the skeleton-based behavior recognition through the common frameworks with GCN and CNN. The double stream of combined score results have been verified its advantages in multiple behavior recognition tasks with double joints stream in research [125], and multiple models of deep feature extractions [122]. The multi-modality human behavior recognition has also been proofed its availability in the increased identification accuracy.…”

A Simple Method to Calculate Positions in Pose Tracking to Verify Work Procedures

“…For example, [7] proposed a deep regression network to estimate absolute 3D coordinates directly from images without intermediate 2D pose representations, and [8] improved the accuracy of poste estimations using a keypoint coordinate representation based on camera line-of-sight. Unlike these approaches, which estimate 3D poses from a single image, methods using multiple images of video frames have also been studied; for example, the authors of [9] used temporal convolutional networks (TCNs) to obtain absolute 3D human poses. While the above methods require a high-end computer with a GPU, the authors of [10] improved MobileNetV2 model [13] designed as a lightweight DNN to enable pose estimation on smartphones.…”