Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors

Abstract

We present a method for inferring diverse 3D models of human-object interactions from images. Reasoning about how humans interact with objects in complex scenes from a single 2D image is a challenging task given ambiguities arising from the loss of information through projection. In addition, modeling 3D interactions requires the generalization ability towards diverse object categories and interaction types. We propose an action-conditioned modeling of interactions that allows us to infer diverse 3D arrangements of humans and objects without supervision on contact regions or 3D scene geometry. Our method extracts highlevel commonsense knowledge from large language models (such as GPT-3), and applies them to perform 3D reasoning of human-object interactions. Our key insight is priors extracted from large language models can help in reasoning about human-object contacts from textural prompts only. We quantitatively evaluate the inferred 3D models on a large human-object interaction dataset and show how our method leads to better 3D reconstructions. We further qualitatively evaluate the effectiveness of our method on real images and demonstrate its generalizability towards interaction types and object categories.

Primitive-Based 3D Human-Object Interaction Modelling and Programming

“…(Huang et al 2022) create new datasets containing RGBD videos and pseudo GT 3D human and rigid object models. With the help of BEHAVE, (Xie, Bhatnagar, and Pons-Moll 2022;Wang et al 2022;Xie, Bhatnagar, and Pons-Moll 2023) get better performance on single-view HOI 3D reconstruction. However, there is still an ignored gap from 2D to 3D, that is humans interacting with articulated objects.…”

IMoS: Intent‐Driven Full‐Body Motion Synthesis for Human‐Object Interactions

“…1, are part of our daily routines. Being able to synthesize such interactions in a virtual 3D environment through textual instructions has widespread applications in several areas, including computer graphics and robotics [ALNM20; HTBT22; WLK*22], movie script visualization [HMLC09] and game design [SSR07]. For instance, in a digitally created movie scene or a virtual role‐playing game, it is natural for the character to interact with the scene objects based on a set of instructions, such as yielding tools, using objects, or eating various items.…”