Dual Networks Based 3D Multi-Person Pose Estimation From Monocular Video

Abstract

Monocular 3D human pose estimation has made progress in recent years. Most of the methods focus on single persons, which estimate the poses in the person-centric coordinates, i.e., the coordinates based on the center of the target person. Hence, these methods are inapplicable for multi-person 3D pose estimation, where the absolute coordinates (e.g., the camera coordinates) are required. Moreover, multi-person pose estimation is more challenging than single pose estimation, due to inter-person occlusion and close human interactions. Existing top-down multi-person methods rely on human detection (i.e., top-down approach), and thus suffer from the detection errors and cannot produce reliable pose estimation in multi-person scenes. Meanwhile, existing bottom-up methods that do not use human detection are not affected by detection errors, but since they process all persons in a scene at once, they are prone to errors, particularly for persons in small scales. To address all these challenges, we propose the integration of top-down and bottom-up approaches to exploit their strengths. Our top-down network estimates human joints from all persons instead of one in an image patch, making it robust to possible erroneous bounding boxes. Our bottom-up network incorporates human-detection based normalized heatmaps, allowing the network to be more robust in handling scale variations. Finally, the estimated 3D poses from the top-down and bottom-up networks are fed into our integration network for final 3D poses. To address the common gaps between training and testing data, we do optimization during the test time, by refining the estimated 3D human poses using high-order temporal constraint, re-projection loss, and bone length regularizations. We also introduce a two-person pose discriminator that enforces natural two-person interactions. Finally, we apply a semi-supervised method to overcome the 3D ground-truth data scarcity. Our evaluations demonstrate the effectiveness of the proposed method and its individual components. Our code and pretrained models are available publicly: https://github.com/3dpose/3D-Multi-Person-Pose.

A Systematic Review of Recent Deep Learning Approaches for 3D Human Pose Estimation

“…[ [50][51][52] Weakly Supervised Learning These methods do not use exact 3D pose annotations; rather, they utilize less precise data like 2D joint locations or multi-view images. The model could be trained using these 2D joint annotations when direct 3D pose labels are not available.…”

“…In another hybrid approach introduced in a subsequent study [52], a fusion network is employed to blend top-down and bottom-up networks, enhancing the robustness of pose estimation from monocular videos. This fusion network unifies the 3D pose estimates to generate the final 3D poses.…”

Advances in Skeleton-Based Fall Detection in RGB Videos: From Handcrafted to Deep Learning Approaches

“…Therefore, there is a need to enhance the keypoint quality through the adoption of advanced pose estimation methods. These newer methods, such as [169], [170] for 2D keypoints, and [171], [172] for 3D keypoints, offer significant advancements in terms of keypoint quality. A list of frequently updated 2D and 3D pose estimation methods can be found online at [173] and [174], respectively.…”

SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video

“…Monocular Multi-Person Reconstruction In contrast to the notable advancements in reconstructing the clothed human for an individual, limited emphasis has been placed on multi-person scenarios, which are evidently more applicable to our daily experiences. Most existing monocular works can only estimate the coarse body shapes of multiple people from monocular observations [4,7,14,19,21,24,25,38,39,46]. Mustafa et al [31] extend prior implicit methods to multiple people and recover spatially coherent 3D human shapes from an RGB image but mainly deal with cases where people are well-spaced and do not interact naturally in close range.…”