Segmentation and semantic labelling of RGBD data with convolutional neural networks and surface fitting

Abstract

We present an approach for segmentation and semantic labeling of RGBD data exploiting together geometrical cues and deep learning techniques. An initial over-segmentation is performed using spectral clustering and a set of NURBS surfaces is fitted on the extracted segments. Then a Convolutional Neural Network (CNN) receives in input color and geometry data together with surface fitting parameters. The network is made of nine convolutional stages followed by a softmax classifier and produces a vector of descriptors for each sample. In the next step an iterative merging algorithm recombines the output of the over-segmentation into larger regions matching the various elements of the scene. The couples of adjacent segments with higher similarity according to the CNN features are candidate to be merged and the surface fitting accuracy is used to detect which couples of segments belong to the same surface. Finally, a set of labeled segments is obtained by combining the segmentation output with the descriptors from the CNN. Experimental results show how the proposed approach outperforms state-of-the-art methods and provides an accurate segmentation and labeling.

Hybrid deep learning and machine learning approach for passive image forensic

“…To prevent overfitting, the augmentation technique with a limited number of training videos were given that improves the accuracy further. Pagnutti et al [16] introduced a technique for the segmentation and semantic labels of RGBD data. After the first step of over‐segmentation using spectral clustering, the extracted blocks fed to the nine convolutional layered neural networks that contain the colour and geometry data.…”

Incremental and Multi-Task Learning Strategies for Coarse-To-Fine Semantic Segmentation

“…Moreover, it should be noticed that, in this case, 29 classes out of 41 derive from just three parent classes, thus confirming the extreme inhomogeneity of this splitting. As done by all the competing approaches (e.g., [16,18,31,32]), we removed both from the prediction and from the evaluation of the results the unlabeled and unknown classes when present. Indeed, they are fictitious classes artificially created during the labeling procedures of the images.…”

“…[39] -42.2 -Hermans et al [40] 54.2 48.0 -Khan et al [41] 58.3 45.1 -Couprie et al [31] 52.4 36.2 -Pagnutti et al [32] 67.2 54.4 -Michieli et al [18] 67.2 54.5 -Eigen et al [16] 75.4 66.9 - Furthermore, we performed an additional incremental step to predict the set of 40 classes starting from the prediction of the set of 13 labels. The results are reported in Table 3.…”

Image-similarity-based Convolutional Neural Network for Robot Visual Relocalization

“…With their rapid development in recent years, neural networks have achieved great success in image classification, (12,13) image retrieval, (14)(15)(16) semantic segmentation, (17)(18)(19) and various applications. (20)(21)(22)(23) Nowadays, convolutional neural networks (CNNs) have also been applied to estimate camera pose from images.…”