Hierarchical Higher-Order Regression Forest Fields: An Application to 3D Indoor Scene Labelling

Abstract

This paper addresses the problem of semantic segmentation of 3D indoor scenes reconstructed from RGB-D images. Traditionally label prediction for 3D points is tackled by employing graphical models that capture scene features and complex relations between different class labels. However, the existing work is restricted to pairwise conditional random fields, which are insufficient when encoding rich scene context. In this work we propose models with higher-order potentials to describe complex relational information from the 3D scenes. Specifically, we relax the labelling problem to a regression, and generalize the higher-order associative P n Potts model to a new family of arbitrary higherorder models based on regression forests. We show that these models, like the robust P n models, can still be decomposed into the sum of pairwise terms by introducing auxiliary variables. Moreover, our proposed higher-order models also permit extension to hierarchical random fields, which allows for the integration of scene context and features computed at different scales. Our potential functions are constructed based on regression forests encoding Gaussian densities that admit efficient inference. The parameters of our model are learned from training data using a structured learning approach. Results on two datasets show clear improvements over current state-of-the-art methods.

Robust and Accurate Plane Segmentation From Point Clouds of Structured Scenes

“…3D point, patches). The graphical model like Conditional Random Fields (CRF) is always employed to capture scene features and different categories (Pham et al, 2015;Vosselman et al, 2017). This encoded 3D contextual information hinders its wide application to construct and optimize such a complex graphical model.…”

MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videos

“…In recent years, we have seen impressive progress in semantic aware reconstruction. Early works [13], [14], [15] use graphical models to assign semantic labels to a geometric reconstruction. SemanticFusion [16] employs a deep network to predict pixel-wise semantic labels given RGB frames, which are then fused into a semantic mapping by leveraging the geometric reconstruction from a RGBD SLAM.…”

SceneCut: Joint Geometric and Object Segmentation for Indoor Scenes

“…State-of-the-art methods such as Mask-RCNN [1], YOLO9000 [3] fail to detect such unknown objects. This behavior is detrimental to the goal of robotic scene understanding that would ideally result in a semantically meaningful map [5]- [8] comprising all objects, environmental structures, and their various complex relations. The ability to extract information about objects (e.g., semantic classes and affordances [9]) and the scene geometry in complex environments under realistic, open-set conditions is increasingly important for robotics.…”

“…In addition, SceneCut reasons jointly over objects and scene geometry and therefore produces a segmentation of non-object surfaces such as ceilings and walls, as well as supporting surfaces such as floor or tabletops. Unlike methods [5], [10]- [12] that deal with geometric segmentation and object segmentation in isolation, our method simultaneously segments objects and planes of a scene in an unified formulation. SceneCut therefore ensures the segmentation of geometric surfaces is consistent with the discovered objects.…”