3D-RelNet: Joint Object and Relational Network for 3D Prediction

Abstract

https://nileshkulkarni.github.io/relative3d/ (b) Results (a) Overview of our method Translation Rotation Scale Instance BBoxes Union (pair) BBox Object Encoder Relative Encoder Final Prediction Image with detections Per object Per pair f Figure 1: (a) Approach Overview: We study the problem of layout estimation in 3D by reasoning about relationships between objects. Given an image and object detection boxes, we first predict the 3D pose (translation, rotation, scale) of each object and the relative pose between each pair of objects. We combine these predictions and ensure consistent relationships between objects to predict the final 3D pose of each object. (b) Output: An example result of our method that takes as input the 2D image and generates the 3D layout. AbstractWe propose an approach to predict the 3D shape and pose for the objects present in a scene. Existing learning based methods that pursue this goal make independent predictions per object, and do not leverage the relationships amongst them. We argue that reasoning about these relationships is crucial, and present an approach to incorporate these in a 3D prediction framework. In addition to independent per-object predictions, we predict pairwise relations in the form of relative 3D pose, and demonstrate that these can be easily incorporated to improve object level estimates. We report performance across different datasets (SUNCG, NYUv2), and show that our approach significantly improves over independent prediction approaches while also outperforming alternate implicit reasoning methods.

Active Vision for Robot Manipulators Using the Free Energy Principle

“…These extensions have also been combined by Lin et al ( 2020 ). 3D-RelNet is also an object-centric model that predicts a pose for each object and their relation to the other objects in the scene (Kulkarni et al, 2019 ). While these approaches seem promising, in their current implementation they only consider video data from a fixed camera viewpoint.…”

“…While a lot of research on learning generative models of the environment has been performed, most of them only consider individual objects (Sitzmann et al, 2019b ; Häni et al, 2020 ), consider scenes with a fixed camera viewpoint (Kosiorek et al, 2018 ; Kulkarni et al, 2019 ; Lin et al, 2020 ) or train a separate neural network for each novel scene (Mildenhall et al, 2020 ; Sitzmann et al, 2020 ). We tackle the problem of an active agent that can control the extrinsic parameters of an RGB camera as an active vision system.…”

Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes From a Single Image

“…The most relevant works to us are [22,46,19,9], which take a single image as input and reconstruct multiple object shapes in a scene. However, the methods [22,46,19] are designed for voxel reconstruction with limited resolution. Mesh R-CNN [9] produces object meshes, but still treats objects as isolated geometries without considering the scene context (room layout, object locations, etc.).…”

“…In indoor environments, object poses generally follow a set of interior design principles, making it a latent pattern that can be learned. By parsing images, previous works either predict 3D boxes object-wisely [14,46] or only consider pair-wise relations [19]. In our work, we assume each object has a multi-lateral relation between its surroundings, and take all in-room objects into account in predicting its bounding box.…”

“…* Corresponding author: hanxiaoguang@cuhk.edu.cn Figure 1: From a single image (left), we simultaneously predict the contextual knowledge including room layout, camera pose, and 3D object bounding boxes (middle) and reconstruct object meshes (right). layout and object locations) for scene reconstruction, but most methods currently adopt depth or voxel representations [39,22,46,19]. Voxel-grid presents better shape description than boxes, but its resolution is still limited, and the improvement of voxel quality exponentially increases the computational cost, which is more obvious in scenelevel reconstruction.…”