Region Merging Driven by Deep Learning for RGB-D Segmentation and Labeling

Abstract

Among the various segmentation techniques, a widely used family of approaches are the ones based on region merging, where an initial oversegmentation is progressively refined by joining segments with similar characteristics. Instead of using deterministic approaches to decide which segments are going to be merged we propose to exploit a convolutional neural network which takes a couple of segments as input and decides whether to join or not the segments. We fitted this idea into an existent iterative semantic segmentation scheme for RGB-D data. We were able to lower the number of free parameters and to greatly speedup the procedure while achieving comparable or even higher results, thus allowing for its usage in free navigation systems. Furthermore, our method could be extended straightforwardly to other fields where region merging strategies are exploited. CCS CONCEPTS• Computing methodologies → Scene understanding; Image segmentation; Neural networks.

HoloLabel: Augmented reality user-in-the-loop online annotation tool for as-is building information

“…Automatic 3D segmentation can be categorized in semantic segmentation, which groups a scene into segments that belong to the same semantic class (e.g., chair or window), and geometric segmentation, which segments the scene into segments with similar geometric properties. Many algorithms perform 3D scene segmentation by processing RGB-D images (Michieli et al 2019, Guo & Hoiem 2013, Gupta et al 2013. Other approaches directly segment 3D visual data, the majority of which operate on 3D point clouds (Poux & Billen 2019, Qi et al 2017.…”

Incremental and Multi-Task Learning Strategies for Coarse-To-Fine Semantic Segmentation

“…Two different CNNs for color and depth and a feature transformation network are exploited in [17]. In [18], a region splitting and merging algorithm for RGB-D data has been proposed. In [19], a MRF superpixel segmentation is combined with a tree-structured segmentation for scene labeling.…”

“…We repeat the same approach also when moving from M 2 to M 3 (i.e., from 15 to 41 classes). This methodology was partially derived by the idea presented in [18] where the softmax information is used for binary classification task. Furthermore, notice that, when training for the finer tasks, the networks corresponding to the coarser ones are frozen, i.e., we do not train in a single step a large size network containing the two (or three) networks for the two (or three) tasks but we perform a set of independent trainings each working on a single stage of the network.…”

“…Moreover, it should be noticed that, in this case, 29 classes out of 41 derive from just three parent classes, thus confirming the extreme inhomogeneity of this splitting. As done by all the competing approaches (e.g., [16,18,31,32]), we removed both from the prediction and from the evaluation of the results the unlabeled and unknown classes when present. Indeed, they are fictitious classes artificially created during the labeling procedures of the images.…”

“…[39] -42.2 -Hermans et al [40] 54.2 48.0 -Khan et al [41] 58.3 45.1 -Couprie et al [31] 52.4 36.2 -Pagnutti et al [32] 67.2 54.4 -Michieli et al [18] 67.2 54.5 -Eigen et al [16] 75.4 66.9 - Furthermore, we performed an additional incremental step to predict the set of 40 classes starting from the prediction of the set of 13 labels. The results are reported in Table 3.…”