Seeing Behind Objects for 3D Multi-Object Tracking in RGB-D Sequences

Müller, Norman; Wong, Yu-Shiang; Mitra, Niloy J.; Dai, Angela; Niesner, Matthias

doi:10.1109/cvpr46437.2021.00601

Cited by 13 publications

(2 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Object reconstruction Most existing work on reconstructing 3D objects from RGB [21,46,53,75,78] and RGBD [45,55,82] data does so in isolation, without the human involvement or the interaction. While challenging, it is arguably more interesting to reconstruct objects in a dynamic setting under severe occlusions from the human.…”

Section: Appearance Modelling: Humans and Objects Without Scene Contextmentioning

confidence: 99%

BEHAVE: Dataset and Method for Tracking Human Object Interactions

Bhatnagar¹,

Xie²,

Petrov³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well as human behavior analysis and human-robot collaboration. This challenging operation scenario requires generalization to vast number of objects, scenes, and human actions. Unfortunately, there exist no such dataset. Moreover, this data needs to be acquired in diverse natural environments, which rules out 4D scanners and marker based capture systems. We present BEHAVE dataset, the first full body humanobject interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them. We record ∼15k frames at 5 locations with 8 subjects performing a wide range of interactions with 20 common objects. We use this data to learn a model that can jointly track humans and objects in natural environments with an easy-to-use portable multi-camera setup. Our key insight is to predict correspondences from the human and the object to a statistical body model to obtain human-object contacts during interactions. Our approach can record and track not just the humans and objects but also their interactions, modeled as surface contacts, in 3D. Our code and data can be found at: http://virtualhumans.mpi-inf.mpg.de/behave.

show abstract

Section: Appearance Modelling: Humans and Objects Without Scene Contextmentioning

confidence: 99%

BEHAVE: Dataset and Method for Tracking Human Object Interactions

Bhatnagar¹,

Xie²,

Petrov³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We compare to competing methods and show that ours can produce better object representations and camera/object trajectories. Note that prior methods often focus on only rigid motion and separated optimization [WLNM21, MWM*21], assume access to geometric priors [MWM*21], a single non‐rigid object with local motion [PSH*21,CFF*22], a global representation [LNSW21,TTG*21, XHKK21], foreground‐background separation and novel‐view rendering without geometric reconstruction [GSKH21, YLSL21, WZT*22, SCL*23], or static scenes with an implicit representation [ZPL*22,SLOD21, YPN*22]. We relax many of these restrictions and demonstrate that our factorized representation naturally enables edits involving object‐level manipulations.…”

Section: Introductionmentioning

confidence: 99%

Factored Neural Representation for Scene Understanding

Wong

Mitra

2023

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

A long‐standing goal in scene understanding is to obtain interpretable and editable representations that can be directly constructed from a raw monocular RGB‐D video, without requiring specialized hardware setup or priors. The problem is significantly more challenging in the presence of multiple moving and/or deforming objects. Traditional methods have approached the setup with a mix of simplifications, scene priors, pretrained templates, or known deformation models. The advent of neural representations, especially neural implicit representations and radiance fields, opens the possibility of end‐to‐end optimization to collectively capture geometry, appearance, and object motion. However, current approaches produce global scene encoding, assume multiview capture with limited or no motion in the scenes, and do not facilitate easy manipulation beyond novel view synthesis. In this work, we introduce a factored neural scene representation that can directly be learned from a monocular RGB‐D video to produce object‐level neural presentations with an explicit encoding of object movement (e.g., rigid trajectory) and/or deformations (e.g., nonrigid movement). We evaluate ours against a set of neural approaches on both synthetic and real data to demonstrate that the representation is efficient, interpretable, and editable (e.g., change object trajectory). Code and data are available at: http://geometry.cs.ucl.ac.uk/projects/2023/factorednerf/.

show abstract