Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images

Abstract

Grounding referring expressions in RGBD image has been an emerging field. We present a novel task of 3D visual grounding in single-view RGBD image where the referred objects are often only partially scanned due to occlusion. In contrast to previous works that directly generate object proposals for grounding in the 3D scenes, we propose a bottom-up approach to gradually aggregate context-aware information, effectively addressing the challenge posed by the partial geometry. Our approach first fuses the language and the visual features at the bottom level to generate a heatmap that coarsely localizes the relevant regions in the RGBD image. Then our approach conducts an adaptive feature learning based on the heatmap and performs the object-level matching with another visio-linguistic fusion to finally ground the referred object. We evaluate the proposed method by comparing to the state-of-the-art methods on both the RGBD images extracted from the ScanRefer dataset and our newly collected SUNRefer dataset. Experiments show that our method outperforms the previous methods by a large margin (by 11.2% and 15.

Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructions

“…It extends 2D visual grounding task to 3D space. Recent researches [1], [3], [8], [13], [20] aim to localize objects in a 3D indoor scenario, which heavily relies on 3D proposals by utilizing PointNet++ [26] or VoteNet [25] as the backbone with increased computational cost. Furthermore, these methods identify objects in an indoor scenario by a 3D bounding box representation, containing 3D location, 3D size and one-axis rotation angle, This 3D bounding box representation has less degree-of-freedom to handle complex manipulation tasks since the orientation is restricted in the horizontal plane.…”