HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation

Abstract

Recovering whole-body mesh by inferring the abstract pose and shape parameters from visual content can obtain 3D bodies with realistic structures. However, the inferring process is highly non-linear and suffers from image-mesh misalignment, resulting in inaccurate reconstruction. In contrast, 3D keypoint estimation methods utilize the volumetric representation to achieve pixel-level accuracy but may predict unrealistic body structures. To address these issues, this paper presents a novel hybrid inverse kinematics solution, HybrIK, that integrates the merits of 3D keypoint estimation and body mesh recovery in a unified framework. HybrIK directly transforms accurate 3D joints to body-part rotations via twist-and-swing decomposition. The swing rotations are analytically solved with 3D joints, while the twist rotations are derived from visual cues through neural networks. To capture comprehensive whole-body details, we further develop a holistic framework, HybrIK-X, which enhances HybrIK with articulated hands and an expressive face. HybrIK-X is fast and accurate by solving the whole-body pose with a one-stage model. Experiments demonstrate that HybrIK and HybrIK-X preserve both the accuracy of 3D joints and the realistic structure of the parametric human model, leading to pixel-aligned whole-body mesh recovery. The proposed method significantly surpasses the state-of-the-art methods on various benchmarks for body-only, hand-only, and whole-body scenarios. Code and results can be found at https://jeffli.site/HybrIK-X/.

GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras

“…the camera. To address the lack of translation, recent methods start to estimate human meshes in the camera coordinates [33,36,53,58,74,77,84,98,106,108,110]. Several approaches recover the absolute translation of the person using an optimization framework [62-64, 80, 107].…”

“…Some approaches approximate the depth of the person using the bounding box size [36,65,110]. HybrIK [53] and KAMA [33] employ inverse kinematics to estimate human meshes with absolute translations in the camera coordinates. Several methods directly predict the absolute depth of each person using a heatmap representation [16,115].…”

SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos

“…In terms of motion representations, human pose estimation can be divided into 2D [8,32,45,52] and 3D [39,46,65,66,68] pose estimation, which output 2D or 3D positions of the target person. Meanwhile, 3D pose estimation can be further divided into model-free and model-based methods, where the latter ones are also referred to as human mesh recovery [9,22,24,29,30,33,37].…”

PoseKernelLifter: Metric Lifting of 3D Human Pose using Sound

“…Given the 3D supervision, the spatial relationship can be directly learned via supervised learning [6,20,55,59]. Various representations have been proposed to effectively encode the spatial relationship such as volumetric representation [43], graph structure [4,11,71,76], transformer architecture [31,34,74], compact designs for realtime reconstruction [37,38], and inverse kinematics [30]. These supervised learning approaches that rely on the 3D ground truth supervision, however, show limited generalization to images of out-of-distribution scenes and poses due to the domain gap.…”