PoseCut: Simultaneous Segmentation and 3D Pose Estimation of Humans Using Dynamic Graph-Cuts

Abstract

We present a novel algorithm for performing integrated segmentation and 3D pose estimation of a human body from multiple views. Unlike other related state of the art techniques which focus on either segmentation or pose estimation individually, our approach tackles these two tasks together. Normally, when optimizing for pose, it is traditional to use some fixed set of features, e.g. edges or chamfer maps. In contrast, our novel approach consists of optimizing a cost function based on a Markov Random Field (MRF). This has the advantage that we can use all the information in the image: edges, background and foreground appearances, as well as the prior information on the shape and pose of the subject and combine them in a Bayesian framework. Previously, optimizing such a cost function would have been computationally infeasible. However, our recent research in dynamic graph cuts allows this to be done much more efficiently than before. We demonstrate the efficacy of our approach on challenging motion sequences. Note that although we target the human pose inference problem in the paper, our method is completely generic and can be used to segment and infer the pose of any specified rigid, deformable or articulated object.

A Robust Stereo Prior for Human Segmentation

“…Larlus and Jurie [12] used a similar approach, using object detection to supply bounding boxes and thus automate the process. More sophisticated priors have been used in works such as ObjCut and PoseCut, which use the output of object detection and pose estimation algorithms respectively to provide a shape prior term [3,8].…”

Detection and Segmentation of Multiple, Partially Occluded Objects by Grouping, Merging, Assigning Part Detection Responses

“…Winn and Shotton 2006;Todorovic and Ahuja 2006;Opelt et al 2006;Shotton et al 2006;Kapoor and Winn 2006;Winn and Jojic 2005;Leibe et al 2004;Pawan Kumar et al 2005) or generative models for pose estimation (e.g. Bray et al 2006).…”

“…Many existing segmentation methods use random field approaches, e.g. the Layout Consistent Conditional Random Field in Winn and Shotton (2006), the Located Hidden Random Field in Kapoor and Winn (2006), the texton based CRF in Shotton et al (2006), the pose-specific MRF in Bray et al (2006), the Pictorial Structure enhanced MRF in Pawan Kumar et al (2005). The inference of the CRF models usually requires loopy belief propagation or sequential tree-reweighted message passing; while graph cut is a widely used solution for inference in the MRF models.…”

PoseField: An Efficient Mean-Field Based Method for Joint Estimation of Human Pose, Segmentation, and Depth

“…For example, Bray et al tackle the problem of human segmentation by introducing a pose-specific MRF, encouraging the segmentation result to look "human-like" [8]. Similarly, Kumar et al use layered pictorial structures to generate an object category specific MRF The first two authors contributed to this work equally as joint first author.…”