Holistic, Instance-level Human Parsing

Abstract

Object parsing -the task of decomposing an object into its semantic parts -has traditionally been formulated as a category-level segmentation problem. Consequently, when there are multiple objects in an image, current methods cannot count the number of objects in the scene, nor can they determine which part belongs to which object. We address this problem by segmenting the parts of objects at an instance-level, such that each pixel in the image is assigned a part label, as well as the identity of the object it belongs to. Moreover, we show how this approach benefits us in obtaining segmentations at coarser granularities as well. Our proposed network is trained end-to-end given detections, and begins with a category-level segmentation module. Thereafter, a differentiable Conditional Random Field, defined over a variable number of instances for every input image, reasons about the identity of each part by associating it with a human detection. In contrast to other approaches, our method can handle the varying number of people in each image and our holistic network produces state-of-the-art results in instance-level part and human segmentation, together with competitive results in category-level part segmentation, all achieved by a single forward-pass through our neural network.

Devil in the Details: Towards Accurate Single and Multiple Human Parsing

“…With multi-scale feature connections and iterative refinement, the parsing and pose tasks boosted each other simultaneously. Considering the practical application, several current works (Li, Arnab, and Torr 2017;Li et al 2017a;Zhao et al 2018) focus on handling the scenario with multiple persons. Usually, it consisted of three sequential steps: object detection (He et al 2017), object segmentation and part segmentation.…”

Graphonomy: Universal Human Parsing via Graph Transfer Learning

“…Mean IoU(%) LIP [14] 59.36 Structure-evolving LSTM [24] 63.57 DeepLab v2 [2] 64.94 Li et al [22] 66.3 Fang et al [12] 67.60 PGN [13] 68.4 RefineNet [30] 68.6 Bilinski et al [1] 68.6 DeepLab v3+ [3] 67.84 Multi-task Learning 68.13 Graphonomy (CIHP) 71.14 Graphonomy (Universal Human Parsing) 69.12 Table 1. Comparison of human parsing performance with several state-of-the-art methods on PASCAL-Person-Part dataset [6].…”

“…The most straightforward solution to universal human parsing would be posing it as a multi-task learning problem, and integrating multiple segmentation branches upon one shared backbone network [2,14,22,25,28]. This line of research only considers the brute-force feature-level information sharing while disregarding the underlying common semantic knowledge, such as label hierarchy, label visual similarity, and linguistic/context correlations.…”

Weakly- and Semi-supervised Panoptic Segmentation

“…This supported their hypothesis that most of the training data for a pixel-level task is statistically correlated within an image, and that randomly sampling a much smaller set of pixels is sufficient. Moreover, [47] and [48] showed improved results by respectively sampling only 6% and 12% of the hardest pixels, instead of all of them, in fully-supervised training. [49] and MCG [50] (b).…”