S$$^2$$Contact: Graph-Based Network for 3D Hand-Object Contact Estimation with Semi-supervised Learning

Tse, Tze Ho Elden; Zhang, Zhongqun; Kim, Kwang In; Leonardis, Aleš; Zheng, Feng; Chang, Hyung Jin

doi:10.1007/978-3-031-19769-7_33

Cited by 11 publications

(8 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given the information of objects, several efforts (Jiang et al 2021;Brahmbhatt et al 2019;Turpin et al 2022) generate the corresponding hand-grasping pose. Furthermore, other approaches focus on refining hand-object grasping state (Grady et al 2021;Yang et al 2021;Tse et al 2022;Taheri et al 2020). For instance, Grady et al (Grady et al 2021) proposed to predict the hand-object contact map and then align the hand with the predicted region.…”

Section: Related Work Hand Object Interactionmentioning

confidence: 99%

“…Many approaches (Grady et al 2021;Tse et al 2022) are proposed to solve the aforementioned problems. To alleviate the potential anatomical irregularities in hand poses, Yang et al (Yang et al 2021) introduced a joint bending constraint to the parametric hand model (Romero, Tzionas, and Black Figure 1: Comparing with TOCH (Zhou et al 2022), our method has three advantages.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling

Hao,

Zhang,

Zhuo

et al. 2024

AAAI

View full text Add to dashboard Cite

Hands are the main medium when people interact with the world. Generating proper 3D motion for hand-object interaction is vital for applications such as virtual reality and robotics. Although grasp tracking or object manipulation synthesis can produce coarse hand motion, this kind of motion is inevitably noisy and full of jitter. To address this problem, we propose a data-driven method for coarse motion refinement. First, we design a hand-centric representation to describe the dynamic spatial-temporal relation between hands and objects. Compared to the object-centric representation, our hand-centric representation is straightforward and does not require an ambiguous projection process that converts object-based prediction into hand motion. Second, to capture the dynamic clues of hand-object interaction, we propose a new architecture that models the spatial and temporal structure in a hierarchical manner. Extensive experiments demonstrate that our method outperforms previous methods by a noticeable margin.

show abstract

Section: Related Work Hand Object Interactionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling

Hao,

Zhang,

Zhuo

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Contact information is often captured as a byproduct in grasp datasets [3,12,32,49] through hand-object proximity or thermal information. Hand-object grasp reconstruction also employs contact to refine the hand and object pose estimation [5,15,20,52,54]. In addition, some works [36,47,62] detect hands and classify their physical contact state into self-contact, person-person contact, and person-object contact.…”

Section: Related Workmentioning

confidence: 99%

Detecting Human-Object Contact in Images

Chen¹,

Dwivedi²,

Black³

et al. 2023

Preprint

View full text Add to dashboard Cite

Humans constantly contact objects to move and perform tasks. Thus, detecting human-object contact is important for building human-centered artificial intelligence. However, there exists no robust method to detect contact between the body and the scene from an image, and there exists no dataset to learn such a detector. We fill this gap with HOT ("Human-Object conTact"), a new dataset of human-object contacts in images. To build HOT, we use two data sources:(1) We use the PROX dataset of 3D human meshes moving in 3D scenes, and automatically annotate 2D image areas for contact via 3D mesh proximity and projection. (2) We use the V-COCO, HAKE and Watch-n-Patch datasets, and ask trained annotators to draw polygons around the 2D image areas where contact takes place. We also annotate the involved body part of the human body. We use our HOT dataset to train a new contact detector, which takes a single color image as input, and outputs 2D contact heatmaps as well as the body-part labels that are in contact. This is a new and challenging task, that extends current foot-ground or hand-object contact detectors to the full generality of the whole body. The detector uses a part-attention branch to guide contact estimation through the context of the surrounding body parts and scene. We evaluate our detector extensively, and quantitative results show that our model outperforms baselines, and that all components contribute to better performance. Results on images from an online repository show reasonable detections and generalizability. Our HOT data and model are available for research at https://hot.is.tue.mpg.de.

show abstract

“…Results on HO3D v2. For HO3D v2, existing methods [34,42,49,30] design various complex strategies via hand-object interaction information to improve the estimation accuracy. For example, HandOccNet [42] carefully designs a network to tackle severe hand occlusion.…”

Section: D Hand Mesh Estimationmentioning

confidence: 99%

“…Non-parametric-based methods [20,15,29] aim to predict the whole mesh vertices directly. More recent work has focused on handhand [39,31,18] and hand-object interactions [52,42,49] which pose new and more complex challenges. In this paper, instead of designing dedicated and resource-intensive heads, we propose a lightweight head that regresses MANO parameters from a pre-trained vanilla ViT for both singlehand estimation and hand-object interaction predictions.…”

Section: Introductionmentioning

confidence: 99%

EgoFish3D: Egocentric 3D Pose Estimation from a Fisheye Camera via Self-Supervised Learning

Liu¹

2022

Preprint

View full text Add to dashboard Cite

Egocentric vision has a wide range of applications for human-centric activity recognition. However, the use of the egocentric fisheye camera allows wide angle coverage but image distortion is introduced along with strong human body self-occlusion, which can impose significant challenges in data processing and model reconstruction. Unlike previous work only leveraging synthetic data for model training, this paper first presents a new real-world EgoCentric Human Action (ECHA) dataset. By using the self-supervised learning under multi-view constraints, we propose a simple yet effective framework, namely EgoFish3D, for egocentric 3D pose estimation from a single image in different real-world scenarios.

show abstract

S$$^2$$Contact: Graph-Based Network for 3D Hand-Object Contact Estimation with Semi-supervised Learning

Cited by 11 publications

References 56 publications

Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling

Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling

Detecting Human-Object Contact in Images

EgoFish3D: Egocentric 3D Pose Estimation from a Fisheye Camera via Self-Supervised Learning

Contact Info

Product

Resources

About