Joint Color and Depth Segmentation based on Region Merging and Surface Fitting

Abstract

The recent introduction of consumer depth cameras has opened the way to novel segmentation approaches exploiting depth data together with the color information. This paper proposes a region merging segmentation scheme that jointly exploits the two clues. Firstly a set of multi-dimensional vectors is built considering the 3D spatial position, the surface orientation and the color data associated to each scene sample. Normalized cuts spectral clustering is applied to the obtained vectors in order to over-segment the scene into a large number of small segments. Then an iterative merging procedure is used to recombine the segments into the regions corresponding to the various objects and surfaces. The proposed algorithm tries to combine close compatible segments and uses a NURBS surface fitting scheme on the considered segments in order to understand if the regions candidate for the merging correspond to a single surface. The comparison with state-of-the-art methods shows how the proposed method provides an accurate and reliable scene segmentatio

Joint segmentation of color and depth data based on splitting and merging driven by surface fitting

“…Again, the surface fitting accuracies before and after the merging are used to evaluate which merge operations are properly joining two parts of the same object. Summarizing, this work extends our two recent conference works, i.e., the region splitting scheme of [2] and the region merging scheme of [5], and combines them into a general segmentation framework that gives better results than each of the two separate methods. In particular by combining the splitting and merging stages it overcomes some limitations of the previous works, e.g., the merging scheme was not able to recover from errors in the initial over-segmentation.…”

“…The fitting accuracy is then used to evaluate the consistency of the segmented regions in order to further split segments not encompassing a single surface in an iterative approach. The work in [5] applies the NURBS fitting scheme within a region merging procedure, starting from an initial over-segmentation and joining adjacent segments on the basis of the fitting accuracy. These two works constitute the starting point for the approach of this paper.…”

“…Following an approach similar to [5,4,33], before entering the proposed iterative clustering algorithm, a nine-dimensional representation of the scene samples is built from the geometry and color data. Firstly the depth and color cameras are jointly calibrated.…”

“…The motivation behind this approach is that segments representing single objects or surfaces are accurately fitted, while segments encompassing multiple surfaces will lead to a poor fitting accuracy. This procedure is followed by a bottom up recursive merging strategy that combines segments belonging to the same object [5]. The algorithm detects the neighboring segments with consistent depth, orientation and color values along the common edges, and attempts to merge them.…”

Unsupervised TOF Image Segmentation through Spectral Clustering and Region Merging

“…Spectral clustering methods are usually used as a first step as image over-segmentation [16]. Therefore, it is necessary to merge the over-segmented regions to obtain the constitutive elements of the image.…”