Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations

“…For example, Text-guided Graph Neural Network [17] conducts instance segmentation on the full scene to create candidate objects as input to a graph neural network [32]; InstanceRefer [39] selects instance candidates from the panoptic segmentation of point clouds; 3DVG-Transformer [40] uses outputs from an object proposal generation module to fully leverage contextual clues for cross-modal proposal disambiguation. The best performing work in this category, BUTD-DETR [20], uses box proposals from a pretrained detector and scene features from the full 3D scene to decode objects with a detection head. The Multi-View Transformer [18] separately models the scene by projecting the 3D scene to a multi-view space, to eliminate dependence on specific views and learn robust representations.…”

“…Modules in NS3D can be trained end-to-end with only the groundtruth referred objects as supervision; each can also be trained individually whenever additional labels are available. In this paper, we use a hybrid training objective similar to prior works [2,20]. Specifically, we use the groundtruth object category to compute a per-object classification loss L oce (applied to all prob c , where c is the category) and the groundtruth final target object to compute a per-expression loss L tce .…”

“…BUTD-DETR [20] 0.670 † 0.530 MVT [18] 0.645 0.584 3DVG-TRANSFORMER [40] 0.514 0.446 INSTANCEREFER [39] 0.480 0.454 TEXT-GUIDED-GNNS [17] 0.450 0.458…”

“…We report experiments on data efficiency compared to four top-performing prior work on ReferIt3D, two objectcentric methods (SAT [37] and TransRefer [16]) and two methods that model the full 3D scene (BUTD-DETR [20] and MVT [18]). We test on 0.…”

“…6,584 examples) in the train set, with the same full test set. We note that BUTD-DETR [20] uses pretrained object classification results on the full ScanNet dataset [13], while others do not. Hence in Table 2, we report NS3D's performance on both settings: pretrained object classification on the full ReferIt3D train set (NS3D + Full), and on the smaller train set only (NS3D).…”