ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis

Abstract

Estimating the articulated 3D hand-object pose from a single RGB image is a highly ambiguous and challenging problem requiring large-scale datasets that contain diverse hand poses, object poses, and camera viewpoints. Most real-world datasets lack this diversity. In contrast, synthetic datasets can easily ensure vast diversity, but learning from them is inefficient and suffers from heavy training consumption. To address the above issues, we propose ArtiBoost, a lightweight online data enrichment method that boosts articulated hand-object pose estimation from the data perspective. ArtiBoost is employed along with a real-world source dataset. During training, ArtiBoost alternatively performs data exploration and synthesis. Art-iBoost can cover various hand-object poses and camera viewpoints based on a Compositional hand-object Configuration and Viewpoint space (CCV-space) and can adaptively enrich the current hard-discernable samples by a mining strategy. We apply ArtiBoost on a simple learning baseline network and demonstrate the performance boost on several hand-object benchmarks. As an illustrative example, with ArtiBoost, even a simple baseline network can outperform the previous start-of-the-art based on Transformer on the HO3D dataset. Our code is available at https://github.com/MVIG-SJTU/ArtiBoost.

Interacting Hand-Object Pose Estimation via Dense Mutual Attention

“…Most previous works tackle 3D hand pose estimation [17,25,40,50,47] and object pose estimation [27,31,44,49] separately. Recently joint hand-object pose estimation has received more focus [14,26,28,12,8,13,11] due to the strong correlation when hands interact with objects. For learning-based methods, Hasson et al [14] propose attraction and repulsion losses to penalize physically implau-sible reconstructions.…”

“…Hasson et al [12] extend to video inputs by leveraging photometric and temporal consistency on sparsely annotated data. To tackle the lack of 3D ground truth, Kailin et al [26] introduce an online synthesis and exploration module to generate synthetic handobject poses from a predefined set of plausible grasps during training. In contrast to the above works, optimization-based methods [13,48,10] formulate the task by firstly estimating initial hand and object poses in isolation, then jointly refining them with contact constraints.…”

“…L is the quantization level. D is the depth radius 1 relative to the wrist joint estimated from the training data, and r z is the wrist joint depth 2 , which is assumed to be known [26,40] to resolve the scale ambiguity in the single view input. Given the camera intrinsic K, pixel coordinates, and depth, we can easily recover the 3D vertex's Euclidean coordinates in the camera space.…”

“…While the former methods [48,13,10] generalize to diverse object classes, the optimization process requires multiple iterations to converge, which is not applicable for real-time applications like XR. In contrast, learning-based methods [26,14,12,8,11] can achieve real-time inference. Motivated by the optimization-based methods, soft contact losses are introduced [14,12] to implicitly guide the network to pursuit plausible hand-object interaction.…”

D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions

“…A key problem in computer vision is to understand how humans interact with their surroundings. Because hands are our primary means of manipulation with the physical world, there has been an intense interest in hand-object pose estimation [5, 14-16, 19, 39, 40] and the synthesis of static grasps for a given object [19,21,25,39]. However, human grasping is not limited to a single time instance, but involves a continuous interaction with objects in order to move them.…”