Scene segmentation from depth and color data driven by surface fitting

Abstract

Scene segmentation is a very challenging problem for which color information alone is often not sufficient. Recently the introduction of consumer depth cameras has opened the way to novel approaches exploiting depth data. This paper proposes a novel segmentation scheme that exploits the joint usage of color and depth data together with a 3D surface estimation scheme. Firstly a set of multi-dimensional vectors is built from color and geometry information and normalized cuts spectral clustering is applied to them in order to coarsely segment the scene. Then a NURBS model is fitted on each of the computed segments. The accuracy of the fitting is used as a measure of the plausibility that the segment represents a single surface or object. Segments that do not represent a single surface are split again into smaller regions and the process is iterated until the optimal segmentation is obtained. Experimental results show how the proposed method allows to obtain an accurate and reliable scene segmentation

Joint Color and Depth Segmentation based on Region Merging and Surface Fitting

“…The scenes have been segmented with the pro- Table 1: Comparison of the performances of the proposed method with (Dal Mutto et al, 2012a) and (Pagnutti and Zanuttigh, 2014). The table shows the average value of the VoI and RI metrics on the six scenes of the dataset made available by the authors of (Pagnutti and Zanuttigh, 2014).…”

“…Approach VoI RI (Dal Mutto et al, 2012a) 2.56 0.84 (Pagnutti and Zanuttigh, 2014) 2.69 0.83 Proposed Method 1.69 0.90 posed method and the obtained results are shown in Fig. 5 while Table 1 presents the numerical results obtained by comparing the data with a manually segmented ground truth.…”

“…5 compares the proposed approach with the methods of (Dal Mutto et al, 2012a), that directly segments the image into the desired number of regions with an approach based on spectral clustering (it can be considered a simplified version of the initial segmentation scheme of Sec. 4) and of (Pagnutti and Zanuttigh, 2014) that exploits a region splitting scheme that recursively partitions each segmented region in two parts. It clearly obtains better performances than the compared approaches.…”

“…The approach has been extended to the segmentation of multiple depth maps in (Srinivasan and Dellaert, 2014). The top down approach (region splitting) has been used in (Pagnutti and Zanuttigh, 2014) where the segmentation is progressively refined in an iterative scheme by recursively splitting the segments that do not represent a single surface in the 3D space. Hierarchical segmentation based on the output of contour extraction has been used in (Gupta et al, 2014), that also deals with object detection from the segmented data.…”

“…et Al. Method Figure 5: Segmentation of some sample scenes with the proposed method and with the approaches of (Dal Mutto et al, 2012a) and of (Pagnutti and Zanuttigh, 2014). The black regions for the proposed approach correspond to samples without a valid depth value from the Kinect that have not been considered for the segmentation.…”