Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture

Abstract

We propose a CNN-based approach for multi-camera markerless motion capture of the human body. Unlike existing methods that first perform pose estimation on individual cameras and generate 3D models as post-processing, our approach makes use of 3D reasoning throughout a multistage approach. This novelty allows us to use provisional 3D models of human pose to rethink where the joints should be located in the image and to recover from past mistakes. Our principled refinement of 3D human poses lets us make use of image cues, even from images where we previously misdetected joints, to refine our estimates as part of an endto-end approach. Finally, we demonstrate how the highquality output of our multi-camera setup can be used as an additional training source to improve the accuracy of existing single camera models.

Learnable Triangulation of Human Pose

“…Such procedure allowed correcting the predictions of 2D joint heatmaps via indirect holistic reasoning on a human pose. In contrast to our approach, in [18] there is no gradient flow from the 3D predictions to 2D heatmaps and thus no direct signal to correct the prediction of 3D coordinates.…”

“…The work [18] used a multi-stage approach with an external 3D pose prior [17] to infer the 3D pose from 2D joints' coordinates. During the first stage, images from all views were passed through the backbone convolutional neural network to obtain 2D joints' heatmaps.…”

Cross View Fusion for 3D Human Pose Estimation

“…Methods Average MPJPE PVH-TSP [27] 87.3mm Multi-View Martinez [16] 57.0mm Pavlakos et al [18] 56.9mm Tome et al [24] 52.8mm Our approach 31.17mm Our approach + MPII 26.21mm and draw the skeletons on the images. Figure 7 shows three estimation examples.…”

U4D: Unsupervised 4D Dynamic Scene Understanding

“…Previous work on 3D pose estimation [46] iteratively builds a 3D model of human-pose consistent with 2D estimates of joint locations and prior knowledge of natural body pose. In [47], multiple cameras are used when estimating the 3D model; this then feeds back into new estimates of the 2D joint locations in each image. This approach allows us to take full advantage of 3D estimates of pose, consistent across all cameras when finding fine grained 2D correspondences between images, and leading to more lifelike, vivid human reconstructions.…”

“…This allows for more robust and complete reconstruction and segmentation. We use a standard set of 17 joints [47] defined as B. A circle C i is placed around the joint position in 2D and a sphere S i is placed around the joint position in 3D based on the confidence map to identify the nearest neighbour vertices for every joint b i .…”