OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

Abstract

Realtime multi-person 2D pose estimation is a key component in enabling machines to have an understanding of people in images and videos. In this work, we present a realtime approach to detect the 2D pose of multiple people in an image. The proposed method uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to learn to associate body parts with individuals in the image. This bottom-up system achieves high accuracy and realtime performance, regardless of the number of people in the image. In previous work, PAFs and body part location estimation were refined simultaneously across training stages. We demonstrate that a PAF-only refinement rather than both PAF and body part location refinement results in a substantial increase in both runtime performance and accuracy. We also present the first combined body and foot keypoint detector, based on an internal annotated foot dataset that we have publicly released. We show that the combined detector not only reduces the inference time compared to running them sequentially, but also maintains the accuracy of each component individually. This work has culminated in the release of OpenPose, the first open-source realtime system for multi-person 2D pose detection, including body, foot, hand, and facial keypoints.Index Terms-2D human pose estimation, 2D foot keypoint estimation, real-time, multiple person, part affinity fields.

Real‐time 3D human pose and motion reconstruction from monocular RGB videos

“…The 2D projections are thereafter grouped into mutually exclusive clusters, and for each cluster, we allocate a representative pose. Then, in the second stage, for an input frame taken from a video stream, we extract in real time its 2D pose using the OpenPose network . The 2D pose is then rescaled so as to be consistent with the 2D projections stored in the database.…”

“…The human poses in these CMU 3D data are represented by n =30 joint positions and rotations. In our work, we use 2D poses that are estimated from an input monocular video using OpenPose (see Section 5 for more details) that are represented by m =14 joint locations (see Figure ). Thus, in order to have a uniform and comparable skeleton, we retarget the CMU .bvh data to a 3D skeleton that its projection in T‐pose matches the 2D skeleton (in T‐pose) returned by OpenPose (see Figure for the new skeleton); the 2D pose projections are then scaled so as their bounding box remains constant over time.…”

“…Most papers in the literature estimate the human pose in two‐dimensional (2D) for one or multiple characters by localizing joint keypoints in pixel space or by extracting the shape silhouette and then retrieving the closest neighbor from a database . More recently, and with the advent of deep learning (DL), the community has been moving to learning‐based discriminative methods, where the effectiveness of 2D human pose estimation has greatly improved . The 3D skeletal reconstruction, though, is a much harder problem .…”

“…To deal with the limitations of the prior work such as the bone length constraints violations, the simultaneously capturing of multiple characters, and the temporal consistency of the reconstructed skeletons, we generate a database with numerous 2D projections by rotating a small angle at a time, the yaw axis of 3D skeletons. Then, we match the input 2D poses (which are extracted from a single video stream using the OpenPose network) with the projections on the database and retrieve the best 3D skeleton pose that is temporally consistent to the skeleton of the previous frames, producing natural and smooth motion.…”

“…[3][4][5] More recently, and with the advent of deep learning (DL), the community has been moving to learning-based discriminative methods, where the effectiveness of 2D human pose estimation has greatly improved. [6][7][8] The 3D skeletal reconstruction, though, is a much harder problem. 9,10 Even though there are methods that are effective at 3D pose reconstruction, they are usually not real-time implementable and suffer from depth and scale ambiguities.…”